Build a Tokenizer Toolkit

Project question: Can you turn inspected text into an exact token sequence, build a stable vocabulary, translate tokens to integer IDs and back, and make every unknown or lossy case visible?

What makes this a project

A tokenizer is a sequence of explicit transformations. In this project you will normalize text under one supplied rule, scan every canonical code point into typed tokens and spans, build a deterministic vocabulary, encode known and unknown tokens, decode IDs, test round trips, and save a tokenizer card another program can inspect.

You are implementing one bounded tokenizer contract. The project does not claim that its normalization, token boundaries, or vocabulary is best for every language, corpus, or model. Keep source identity and transformation evidence visible at every stage.

You will produce:

• versioned configuration tied to one inspected corpus;
• canonical text with recorded changes from analysis text;
• complete non-overlapping token records and spans;
• a deterministic vocabulary with fixed reserved IDs;
• encoded sequences and explicit unknown-token traces;
• decoded token sequences and honest round-trip results;
• numerical summaries, figures, artifacts, and separate replay evidence; and
• a concise tokenizer card whose claims remain within this contract.

What you should know first

The project follows the Python path through Chapter 13. You should be able to scan strings, classify code points under supplied rules, maintain ordered records and counts, validate IDs, work across files, save JSON and CSV artifacts, and test deterministic transformations. TEXT-01 is useful but optional; a canonical inspected-corpus fixture supplies the same starting boundary when it was skipped.

Supplied inputs

Use accepted document records and versioned corpus identity from TEXT-01 or the supplied fallback, plans/tokenizer_config.json, a tiny hand-check corpus, and separate unseen-text fixtures. The configuration records normalization and scanner versions, reserved-symbol order, maximum vocabulary size, and boundary-token policy.

Every output retains document_id and input corpus-manifest identity. Vocabulary construction uses all accepted documents in this bounded Python project. Later subjects define train, validation, test, sampling, and leakage policies.

Normalization and scanning

Start from TEXT-01 analysis text, whose line boundaries are already \n. Create canonical text from left to right by:

1. preserving every \n exactly;
2. replacing each non-empty run of other Unicode whitespace with one ASCII space; and
3. preserving every other code point and its case exactly.

Do not trim leading or trailing space, lowercase text, remove punctuation, replace accents, or apply Unicode normalization. Record whether canonical text differs and how much whitespace collapsed. Never present canonical text as the unmodified source.

Scan canonical text completely:

• each \n becomes one newline token;
• each ASCII space becomes one space token;
• each maximal run of Unicode alphanumeric code points or underscore becomes one text token; and
• each remaining code point becomes one symbol token.

Each token record contains text, kind, zero-based position, and half-open span [start, end). Adjacent spans touch, the first begins at zero, the last ends at canonical-text length, and joining token text reconstructs canonical text. Empty text produces an empty sequence.

Vocabulary and codec

The fixed ordered reserved symbols are:

<pad>  <unk>  <bos>  <eos>

Their IDs are their configured positions from zero. Reject repeated reserved strings, changed order, or a vocabulary limit smaller than the reserved count. Scanner output cannot collide with these strings because angle brackets are separate symbol tokens.

Count scanned tokens across documents in manifest order. After the reserved symbols, order ordinary tokens by decreasing corpus frequency, then lexicographically increasing token text for exact ties. Truncate ordinary tokens to the configured maximum vocabulary size. IDs are contiguous. Space and newline are ordinary tokens and need escaped human-readable displays.

Encoding may add exactly one <bos> and <eos> around each non-empty document according to configuration. Replace an ordinary token absent from the vocabulary with <unk> and record document ID, token position, text, kind, and span. Report unknown count and fraction with token-count denominator. For an empty sequence, save the fraction as JSON null and display not defined.

Decoding validates non-boolean integer IDs in vocabulary range and returns the exact vocabulary token sequence. It preserves reserved symbols by default. A document without reserved or unknown symbols reconstructs canonical text by joining decoded token text. A validated helper may remove exactly one configured <bos>/<eos> pair. <unk> cannot recover the source token; never guess it.

The project workspace

project/
  README.md
  data/                              # supplied, read only
  plans/tokenizer_config.json        # supplied starting config
  src/config.py                      # reader implementation
  src/normalize.py                   # reader implementation
  src/scan.py                        # reader implementation
  src/vocabulary.py                  # reader implementation
  src/codec.py                       # reader implementation
  src/report.py                      # reader implementation
  src/main.py                        # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py              # supplied, read only

Keep normalization, scanning, vocabulary construction, the integer codec, and reporting independently inspectable. Maintain one continuous workspace through all milestones.

Required evidence

The completed project contains:

• versioned tokenizer_config.json;
• vocabulary.json and a readable escaped vocabulary table;
• per-document token records with kinds and spans;
• per-document encoded ID sequences;
• unknown_tokens.csv and document/corpus summaries;
• token-frequency-rank and per-document token/unknown-count figures with exact plot data;
• normalization, scanning, vocabulary-order, codec, and round-trip fixtures;
• tokenizer_manifest.json and a replay agreement or mismatch record; and
• a concise tokenizer card at report.md.

Checks should include empty text; spaces, tabs, multiple newlines, leading/trailing whitespace, punctuation, underscores, non-ASCII letters, emoji, combining marks; span coverage and reconstruction; reserved collision attempts; repeated reserved strings; too-small and truncated vocabularies; equal-frequency ties; unseen text; invalid, negative, boolean, and out-of-range IDs; missing or doubled boundaries; input immutability; artifact read-back; replay; and prose review that rejects claims of universal, linguistic, compression, or model-quality superiority.

Limits

This project does not choose the best tokenizer or define natural-language words. It does not introduce locale-specific segmentation, Unicode normalization, stemming, lemmatization, stop-word removal, byte fallback, production BPE or Unigram training, stochastic tokenization, corpus sampling, train/test leakage policy, parallel execution, compression benchmarks, embedding lookup, language-model training, or claims about downstream model quality. The optional ordered-pair merge extension is separate from Core completion.

Review

The final review asks whether every transformation preserves document and corpus identity, normalization changes are explicit, token spans cover canonical text, vocabulary order is deterministic, unknowns remain traceable, decoding keeps reserved symbols, round-trip limits are honest, replay is separate, and the tokenizer card states exactly what the toolkit can and cannot support.