Milestone 6 of 11

Parse defensively and rebuild the index

Check bounds before iteration or slicing, consume the payload exactly, reject malformed streams, and recover all positional records.

Decoding untrusted bytes is a boundary exercise. Check each count and length before using it, then rebuild the complete source index without silent repair.

Goal

Parse the bounded payload defensively, restore stable document IDs and absolute positions, reject malformed structure, and construct the complete positional inverted index.

Inputs

Use the payload bytes, validated configuration and bounds, canonical integer decoder, strict UTF-8 rule, document table, and source identity. Parse the exact ordered stream of document count and IDs, term count and terms, posting count, document gaps, position counts, and position gaps.

Before looping, slicing, or allocating from any declared count or length, check its configured bound and the remaining payload. IDs and terms must decode strictly as UTF-8; IDs are non-empty and unique; terms are non-empty, unique, and strictly lexicographically increasing. Every term and posting count is positive where required. Consume the payload exactly and reject trailing bytes.

Deliverables

Implement defensive parsing in src/payload.py and reconstruction in src/rebuild.py. Produce output/rebuilt_inverted_index.json with exact term text, lexicographic term order, manifest document order, positive counts, and full-token positions. Preserve source and configuration identity in the rebuild record.

Checks

Check empty and zero-term documents, one and several postings, repeated terms, zero and large first values, and exact case, non-ASCII, and underscore terms. Reject absent or unterminated integers, non-canonical integers, excessive values, ordinal overflow, invalid counts, duplicate or out-of-order IDs or terms, invalid UTF-8, truncated fields, over-bound counts or lengths, non-increasing reconstructed ordinals or positions, and payload trailing bytes.

Check every decoded ID and position agrees with the source table and gap trace, every posting's position count equals its positive term count, and no parser operation reads outside the declared payload or mutates input bytes.

Workspace

Keep bounded field parsing in src/payload.py and index construction in src/rebuild.py. Write the rebuilt index and parser rejection evidence. Do not validate the outer container or compare searches yet.

Hints

HintCheck before repetition

Read a count, check its bound and the remaining bytes, then enter its loop. Never allocate a list from an unchecked declared count.

HintAccumulate gaps

Start each posting from zero, add each decoded gap, and check strict increase after the first value. Resolve the final ordinal through the stored document table.

HintExact consumption matters

After the final position, the parser must be at the declared payload end. Remaining bytes are malformed structure, not harmless padding.

Review

Inspect a payload that declares one extra term but ends early. Which check should fail before the parser attempts to read the term bytes, and why is a partial rebuilt index unsafe?

How to check your work

Checks compare rebuilt records and malformed-stream outcomes with the fixtures. The reference rejects at the first defensible boundary and never sorts or repairs a malformed payload into a plausible index.