Milestone 8 of 11

Prove structural index equivalence

Rebuild exact document and posting structures and compare deterministic serializations with first-mismatch evidence.

Rebuilding an index can preserve answers while changing its representation, or look identical while losing one occurrence. Test structure directly before testing search behavior.

Goal

Re-run TEXT-04 invariants, compare the source and rebuilt indexes exactly, and retain the first field-level mismatch when structural equality fails.

Inputs

Use the accepted source index, rebuilt_inverted_index.json, document table, source identities, and deterministic serialization rules. The rebuilt index must retain exact term text, lexicographic term order, document IDs in manifest order, positive term counts, and full-token positions.

Deliverables

Implement structural comparison in src/equivalence.py. Produce an exact structural comparison record containing source and rebuilt identities, document and term counts, posting and occurrence totals, invariant results, deterministic serialization digests or bytes, and an agreement or first-mismatch path with expected and observed values.

Checks

Re-run every TEXT-04 posting, document-length, and corpus-count invariant. Check empty and zero-term documents, repeated terms across documents, exact case, non-ASCII and underscore terms, document and term order, positive counts, and all full-token positions. Deterministic serialization of source and rebuilt indexes must equal byte for byte.

Deliberately remove, reorder, or alter one posting in a separate fixture and confirm the first exact mismatch is retained. Structural equality and search equivalence are separate evidence; do not waive one because the other passes.

Workspace

Keep invariant checks and comparison in src/equivalence.py. Write only the structural comparison record and mismatch fixtures. Do not compare query results yet.

Hints

HintCompare identity before counts

A matching count can hide a different document. Check source identity, order, and text before aggregate totals.

HintUse one canonical serialization

Serialize both indexes with the same deterministic order and schema, then compare bytes. Keep the field-level comparison to explain a mismatch.

HintKeep the first mismatch

Walk records in published order and stop at the first difference, naming its path and values. A later mismatch is less useful if an earlier one already explains the failure.

Review

Choose one repeated term and trace its source and rebuilt postings through the invariants. Which mismatch would a total-count check miss?

How to check your work

Checks compare invariants, serializations, and first-mismatch records with the source and rebuilt fixtures. The supplied fixture proves exact structure rather than relying on a successful decode alone.