Verify corpus, tokenizer, index, configuration, and queries

Before deriving positional state, prove that the source records, tokenizer, index, configuration, and queries belong together. A plausible-looking result from incompatible inputs answers a different question.

Goal

Validate corpus and tokenizer identities, complete scanner-token positions, TEXT-04 posting compatibility, configuration rules, and every structured query before the engine derives new state.

Inputs

Read accepted documents and complete scanner-token records in corpus-manifest order, their corpus and tokenizer identities, and the TEXT-04 positional index or canonical equivalent. Read plans/phrase_proximity_config.json, plans/queries.jsonl, and the tiny fixtures.

The configuration names the supported query kinds, selects text as the only indexable token kind, sets a non-negative excerpt radius, permits only after and either directions, and records the tokenizer, corpus, index, and rule versions. A query has a unique non-empty query_id and is exactly one of:

{"query_id":"q1","kind":"phrase","text":"small language model"}
{"query_id":"q2","kind":"proximity","left":"data","right":"model","direction":"after","maximum_intervening_terms":3}

Keep complete scanner positions, spans, exact token text, and empty or zero-searchable-term documents in accounting. Do not normalize input into a different rule while checking it.

Deliverables

Implement the identity and compatibility checks in src/config.py and the document/index checks in a clearly separated loader. Produce an ordered, validated input record containing document IDs, manifest positions, identity versions, complete token counts, searchable-term counts, query identity, and the accepted rules.

Checks

Reject changed corpus or tokenizer identity, missing or duplicate document IDs, changed manifest order, non-zero-based or duplicate or missing full-token positions, overlapping or gapped spans, spans outside canonical text, and token spans that do not reconstruct the canonical source. Reject altered token text, unknown token kinds, a changed TEXT-04 indexable-term rule, and TEXT-04 postings whose documents, terms, counts, positions, or ordering disagree. A document with no text records remains with an empty term stream.

Reject unknown configuration or rule versions, unsupported query kinds, an indexable kind other than text, a negative excerpt radius, unsupported directions, duplicate query IDs, malformed records, extra fields that change the accepted shape, and a non-boolean or negative proximity limit. Check punctuation-only and ignored-only fields, repeated text, case differences, non-ASCII text, underscores, and empty or zero-term documents. Confirm that supplied inputs are not mutated.

Workspace

Keep identity, configuration, document, index, and query-shape checks in this milestone. Do not derive a new positional view, match queries, build excerpts, or save figures yet.

Hints

Review

Follow one accepted document from its manifest entry through its token records and TEXT-04 postings. Then follow one rejected fixture to the exact rule that rejects it. Why would silently applying a new tokenizer make a later match untrustworthy?

How to check your work

Checks compare identities, positions, posting records, configuration, and query shapes with the supplied fixtures. The supplied fixture rejects incompatible inputs rather than guessing or repairing them.