Milestone 3 of 10

Implement and trace the bounded encoder

Emit the longest known byte phrase, grow consecutively below the limit, stop exactly at the bound, and retain complete source-range evidence.

The encoder grows phrases only when the next phrase is known. Keep each emitted code tied to the source bytes and dictionary state that produced it.

Goal

Implement the bounded encoder, retain its complete trace, stop growth exactly at the dictionary limit, and prove emitted phrases concatenate to the source bytes.

Inputs

Use validated bytes, the 256-entry initial dictionary, and a configured max_dictionary_size. Follow this rule:

keep the longest current phrase already in the dictionary;
extend it when appending the next byte produces a known phrase;
otherwise emit the current code, add the extended phrase at the next code when below the limit, and begin with the next byte; and
emit the remaining current phrase at the end.

Every non-empty input byte belongs to exactly one emitted phrase. Dictionary entries are never removed, replaced, or reset inside a document.

Deliverables

Implement encoding in src/dictionary.py. Produce output/encoder_trace.jsonl with source half-open byte range, current phrase, next byte where applicable, emitted code, attempted phrase, whether it was added, assigned code, and dictionary size after each step. Produce primary output/document_codes.jsonl with source identity, code order, and final dictionary size.

Checks

Check empty, one-byte, repeated, non-repeated, newline, punctuation, underscore, and multi-byte byte inputs. Check one-growth and full-dictionary limits, growth stopping exactly at the limit, longest-known-phrase selection, consecutive codes, source-range coverage, and phrase-concatenation reconstruction.

Verify empty input emits no codes and final size 256; every non-empty source byte belongs to one emitted phrase; and no dictionary state crosses documents. Do not use a character dictionary, variable-width code, reset heuristic, or end marker.

Workspace

Keep encoder state and trace construction in src/dictionary.py. Read source and dictionary helpers without mutation. Do not implement decoder recovery yet.

Hints

HintTrack the current phrase

Keep a byte-tuple phrase and test the extended tuple before emitting. The failed extension is the attempted new phrase, not part of the emitted current phrase.

HintLimit before adding

Emit the current code either way, but add the new phrase only while the dictionary size is below the configured limit.

HintProve coverage by ranges

Sort or inspect trace ranges and verify they tile [0, len(source_bytes)) without gaps or overlaps.

Review

Trace a repeated two-byte input by hand. Which phrase is longest at each step, when is a new code assigned, and how do emitted phrases reconstruct the bytes?

How to check your work

Checks compare encoder codes, additions, ranges, final sizes, and reconstructed bytes with the fixtures. The supplied fixture follows the bounded longest-phrase rule and does not claim a standard codec.