Milestone 8 of 8

Write and audit the tokenizer card

State the exact rules, data boundary, reserved IDs, unknown behavior, round-trip limits, artifacts, replay, and unsupported claims.

The tokenizer card describes the transformation another reader or program will receive. It should make every lossy boundary and identity visible.

Goal

Write and audit a concise tokenizer card covering normalization and scanning, vocabulary source and order, reserved IDs, unknown behavior, round-trip limits, corpus scope, artifacts, and unsupported claims.

Inputs

Use the tokenizer configuration and manifest, canonical and token records, vocabulary, encoded sequences, unknown traces, summaries, fixtures, figures, and replay record. Read saved artifacts rather than relying on current editor state.

Deliverables

Complete project-level report.md with these sections:

  1. Identity and scope: state corpus and tokenizer identity, document order, and the bounded corpus used to build vocabulary.
  2. Normalization and scanning: state exact whitespace, newline, case, punctuation, code-point, token-kind, and span rules.
  3. Vocabulary and IDs: state reserved order, maximum size, frequency and lexical ordering, truncation, and escaped space/newline display.
  4. Encoding and decoding: state boundary behavior, unknown traces, null fractions, ID validation, reserved-symbol preservation, and exact round-trip conditions.
  5. Evidence and limits: link artifacts, figures, fixtures, and replay, then state what the toolkit cannot claim about languages, compression, or model quality.

Audit each example against a saved record. Make clear that canonical text is a transformed representation and that an <unk> sequence cannot reconstruct its unknown source token.

Checks

Check that the card states every required rule and fixed reserved ID, names the source corpus and vocabulary order, and agrees with unknown and round-trip fixtures. Follow one known and one unknown token through the artifacts.

Check that the card states at least one concrete limitation and contains no claim of universal segmentation, linguistic word boundaries, compression superiority, or downstream model quality. Confirm replay claims match the saved record.

Workspace

Keep the tokenizer card at project-level report.md. Do not duplicate full token or vocabulary files in prose. The card explains durable artifacts; the configuration, records, vocabulary, codec fixtures, figures, manifest, and replay remain primary evidence.

Hints

HintFollow one transformation
Choose a source excerpt and point to its canonical text, spanned tokens, IDs, decoded tokens, and reconstruction status.
HintName loss where it happens
Whitespace collapse and unknown replacement are different boundaries. State each separately rather than calling the whole tokenizer lossless or lossy.

Review

Read the card as a later project author. Can they reproduce the rules, preserve identities, recognize unknowns, and know when reconstruction is impossible? Replace broad praise with an exact artifact-backed statement or remove it.

How to check your work

Checks compare the card structure and evidence links with the supplied fixture. The supplied fixture is an exemplar of bounded documentation, not a claim that this tokenizer is best.