Milestone 5 of 8
Encode known and unknown text
Add configured boundaries, replace absent tokens with a visible unknown ID, retain complete traces, and report fractions with explicit denominators.
Encoding maps token records to vocabulary IDs. Unknown tokens must remain
visible as evidence rather than disappearing behind <unk>.
Goal
Encode known and unseen token sequences, add configured boundary symbols, replace out-of-vocabulary tokens explicitly, and calculate unknown counts and fractions with visible denominators.
Inputs
Use token records, the validated vocabulary, and the boundary-token policy.
For a non-empty document, optionally add exactly one <bos> before and one
<eos> after the ordinary token IDs. An empty token sequence remains empty.
For each ordinary token missing from the vocabulary, emit the <unk> ID and
record document ID, token position, exact text, kind, and canonical-text span.
Do not mutate the token record or vocabulary.
Deliverables
Implement encoding in src/codec.py and save per-document ID sequences,
output/unknown_tokens.csv, and document/corpus summaries. Include original
token count, unknown count, unknown fraction, boundary policy, and tokenizer and
corpus identity.
For an empty token sequence, save unknown fraction as JSON null and display it
as not defined. Keep denominator fields beside every defined fraction.
Checks
Encode known text, unseen words, unseen symbols, a truncated-vocabulary token, and empty text. Check exact boundary count and positions, known token IDs, unknown replacements, trace fields, and denominator calculations.
Confirm that an empty document receives no invented boundary pair, that every
<unk> in an encoded ordinary position has a trace, and that input tokens and
vocabulary remain unchanged. Use document identity to prevent unknown traces
from being paired with another document.
Workspace
Keep encoding and unknown tracing in src/codec.py; use vocabulary mappings
from src/vocabulary.py. Write encoded records, summaries, and
unknown_tokens.csv under output/.
Hints
HintTrace before replacing
<unk> ID.HintBoundary IDs are not ordinary tokens
<bos> and <eos> according to the document policy after encoding ordinary tokens. Do not count them as unknown-token denominator entries.Review
Follow one unknown token from canonical span to token record, unknown trace,
and encoded ID. Explain why <unk> preserves the fact that something was
unknown but cannot preserve its original text by itself.
How to check your work
Checks compare boundary behavior, unknown traces, and zero-denominator handling with the supplied fixture. The supplied fixture never hides an out-of-vocabulary token.