Milestone 1 of 11

Verify source-index identities, configuration, and bounds

Check exact source identities, positional-index structure, query compatibility, format versions, and positive non-boolean parser bounds.

Binary data is only useful when its source and limits are known. Before encoding, validate every identity, ordering rule, format version, and bound.

Goal

Verify the source positional index, corpus, tokenizer, document and query identity, compression configuration, format version, ordering rules, and all positive decoding limits before deriving bytes.

Inputs

Read the accepted TEXT-04 corpus, tokenizer identity, manifest, complete positional inverted index, queries, expected search artifacts, or the version-matched fallback. Read plans/compression_config.json and malformed fixtures.

The configuration supplies format identity, accepted container version, strict UTF-8 and canonical integer-code rules, deterministic ordering, numerical size-accounting rules, and positive bounds for payload bytes, documents, document-ID bytes, terms, term bytes, postings per term, positions per posting, total postings, total positions, encoded integer bytes, and decoded integer value.

Deliverables

Implement compatibility checks in src/config.py and clearly separated loaders. Produce one ordered validated record carrying all source identities, document and term order, query and result configuration, format versions, integer rule, UTF-8 rule, bounds, and size-accounting policy.

Checks

Reject changed corpus, tokenizer, index, query, or configuration identity; changed document, term, posting, or position order; missing or duplicate IDs; altered token selection, term spelling, counts, or positions; unsupported format versions; and non-positive, boolean-like, or missing limits. Check empty and zero-term documents, zero and large first ordinals and positions, repeated terms across documents, exact case, non-ASCII, and underscore terms.

Check that source scoring, ranking, result limits, and excerpt rules are carried unchanged. Confirm input artifacts are not mutated and that every malformed fixture is assigned to a later parser check rather than silently accepted.

Workspace

Keep source, identity, configuration, and bound checks in src/config.py and its loaders. Do not derive gaps, encode integers, write payloads, or create figures yet.

Hints

HintIdentity travels with bytes

Copy every source and rule identity into the validated record. A payload can decode correctly and still describe a different index.

HintBounds precede loops

Validate each configured positive bound before a later module uses it to allocate or iterate.

HintKeep zero-term documents

A document with no searchable terms is valid source accounting. Do not delete it because it has no postings.

Review

Follow one document from the source manifest through its index records and configuration identity. Then inspect an incompatible fixture. Which later claim would become unsafe if the index identity changed silently?

How to check your work

Checks compare identities, ordering, versions, rules, and limits with the fixtures. The supplied fixture rejects incompatible source material rather than repairing it.