Milestone 4 of 8

Build a deterministic vocabulary

Reserve fixed IDs, count scanner tokens, apply exact frequency and lexical ties, enforce the size limit, and verify inverse mappings.

A vocabulary gives stable IDs only when its source, ordering, and size limit are explicit.

Goal

Count scanned tokens, reserve fixed IDs, apply frequency and lexicographic tie rules, enforce the vocabulary limit, and verify contiguous forward and inverse mappings.

Inputs

Use all token records in corpus-manifest document order and the validated configuration. Reserve IDs zero through three for <pad>, <unk>, <bos>, and <eos>. Count ordinary token text across all accepted documents.

Order ordinary tokens by decreasing corpus frequency, then lexicographically increasing token text when counts tie. Keep only as many as fit the configured maximum vocabulary size. Space and newline are ordinary tokens.

Deliverables

Implement vocabulary construction in src/vocabulary.py and save output/vocabulary.json plus a readable escaped table. Each entry includes token text, integer ID, corpus count, token kind where unambiguous, and reserved status. IDs must be contiguous.

Save the vocabulary source corpus identity, configuration identity, ordering rule, maximum size, and count of excluded ordinary token types.

Checks

Use equal-frequency tokens to verify lexicographic ties, a vocabulary that truncates ordinary tokens, a vocabulary containing space and newline, and a corpus smaller than the size limit. Reject repeated reserved strings, changed required order, and too-small maximum size.

Check reserved IDs, contiguous ordinary IDs, token-to-ID and ID-to-token inversion, exact corpus counts, deterministic output under the same ordered records, and no mutation of tokens or configuration.

Workspace

Keep counting, ordering, and mappings in src/vocabulary.py. Read token records from src/scan.py; write vocabulary artifacts under output/. Do not encode documents in this milestone.

Hints

HintUse a two-part sort key

For an ordinary token, sort by negative count first and token text second.

HintEscape only the display

The machine-readable vocabulary keeps exact space or newline text. The readable table can show \\s or \\n with a documented display rule.

Review

Choose two equal-frequency tokens and show why their order is stable. Trace one ordinary ID back to its count, token text, and corpus identity.

How to check your work

Checks compare reserved IDs, ordinary ordering, truncation, and inverse mappings with the supplied fixture. The supplied fixture does not claim that frequency order is universally best.