Milestone 5 of 10

Prove encoder and decoder agreement

Check normal, empty, repeated, special-code, full-dictionary, and malformed streams while comparing bytes, codes, traces, and final dictionary sizes.

Matching code streams are not enough: compare the two state machines and the bytes they produce, including the special case and failures.

Goal

Prove encoder and decoder agreement on normal, special-code, full-dictionary, empty, repeated, and malformed streams, including output, codes, traces, and final dictionary sizes.

Inputs

Use source bytes, encoder and decoder traces, configured dictionary limits, fixed-width code values, and deliberately malformed streams. For each accepted document, compare encoder codes with decoder input, emitted and recovered byte phrases, dictionary additions and codes, output ranges, and final sizes.

Deliverables

Implement the comparison in src/codes.py or a clearly separated helper. Produce agreement records for every source and limit, with byte equality, code equality, trace agreement, final-size agreement, and strict-text readiness. Produce malformed-stream records with first invalid code position and reached dictionary size.

Checks

Hand-check empty, one-byte, repeated, non-repeated, non-ASCII, newline, punctuation, special next-code, one-growth, ordinary, and 65536-entry limits. Check growth stopping at the limit, invalid first/existing/next codes, negative values before packing, above-two-byte values, and malformed streams.

Verify successful recovered bytes equal source bytes exactly, encoder and decoder final dictionary sizes agree, code order is unchanged, and a failed decode is never reported as a recovered document. Keep input bytes and traces immutable.

Workspace

Keep agreement and malformed results in the codec workspace. Read encoder and decoder traces without mutation. Do not pack payload fields or wrap containers yet.

Hints

HintCompare state, not just output

Two implementations can produce the same bytes while adding different phrases. Compare each trace addition and final dictionary size.

HintKeep failure separate

An invalid stream may have a partial output buffer, but it is not a successful recovery. Store its first invalid position and failure state separately.

HintUse limits as fixtures

Run the same source at one-growth and full-dictionary limits. The code stream may change while every successful recovery remains exact.

Review

Choose a special-code example and compare the encoder's attempted phrase with the decoder's reconstructed phrase. Which state equality proves the match?

How to check your work

Checks compare agreement fields, traces, final sizes, and malformed outcomes with the fixtures. The supplied fixture requires exact bytes and state, not merely a matching length.