Milestone 6 of 8

Decode and test round trips

Reject invalid IDs, preserve special symbols by default, reconstruct known canonical text, and expose why unknown source text cannot be recovered.

Decoding inverts vocabulary IDs, not unknown source text. Test exact cases and make the non-invertible boundary explicit.

Goal

Validate IDs, preserve special symbols by default, prove token/ID inversion, reconstruct known canonical text, and demonstrate why text represented by <unk> cannot be recovered.

Inputs

Use the versioned vocabulary and encoded sequences. Every ID must be a non-boolean integer within vocabulary range. decode_ids returns exact vocabulary token text, including <pad>, <unk>, <bos>, and <eos> unless a caller requests one documented boundary-symbol view.

A separate validated helper may remove exactly one configured <bos>/<eos> pair before reconstruction. It must reject missing, doubled, reversed, or internal boundary symbols rather than silently changing the sequence.

Deliverables

Implement decoding and reconstruction in src/codec.py. Save fixtures for:

  • token-to-ID-to-token inversion;
  • known canonical-text reconstruction without reserved or unknown symbols;
  • configured boundary-pair removal and reconstruction; and
  • an unknown-containing sequence whose original text is explicitly unrecoverable.

Keep decoded token sequences and reconstruction status separate. Never insert guessed text for <unk> or silently remove <pad>.

Checks

Reject negative, boolean, non-integer, and out-of-range IDs. Check reserved IDs are preserved by default and ordinary token IDs invert exactly. Verify known canonical reconstruction with and without one valid boundary pair.

Use missing, doubled, reversed, and internal boundary fixtures. Confirm an unknown sequence decodes to <unk> and reports non-reconstructable source text. Check encoded sequences and vocabulary remain unchanged.

Workspace

Keep decoding, boundary validation, and reconstruction in src/codec.py. Write round-trip fixtures and statuses under output/; do not change the vocabulary to make an unseen text round trip.

Hints

HintDecode before reconstructing
First map every ID to its exact vocabulary token. Then decide whether the resulting sequence satisfies a reconstruction boundary.
HintBoolean is not an ID here
Python treats booleans as integers in some contexts. Reject them explicitly so True cannot silently become ID 1.

Review

Compare a known round trip and an unknown-containing sequence. Explain which information was preserved, which was replaced, and why decoding cannot recover the unknown source text.

How to check your work

Checks compare ID validation, reserved-symbol preservation, boundary checks, and reconstruction states with the supplied fixture.