Verify source, tokenizer, and search configuration

Before indexing anything, prove that the source records and their tokenizer belong together. A search index built from incompatible artifacts may look reasonable while answering a different question.

Goal

Validate corpus and tokenizer identities, document order, complete token records, source spans, the indexable-kind rule, query records, and positive search limits.

Inputs

Read the accepted documents and TEXT-02 token records in manifest order, the versioned tokenizer configuration, vocabulary identity, plans/search_config.json, plans/queries.jsonl, and the tiny fixtures. The search configuration must select text tokens, set a positive top_k and excerpt radius, and name the ranking-rule version. Queries have unique query_id values and may be empty, repeated, punctuation-only, case-different, or partly missing.

A token record has text, kind, zero-based position, and a half-open span. Its spans touch from the beginning to the end of canonical text. Keep spaces, newlines, punctuation, and symbols in source accounting even though only text records enter the index.

Deliverables

Implement validation in src/config.py and src/documents.py (or clearly separated loaders). Produce ordered records containing document ID, manifest position, corpus/tokenizer identity, canonical length, token count, indexable length, search limits, and query identity.

Checks

Reject changed corpus or tokenizer identity, duplicate or missing document IDs, changed manifest order, duplicate or missing positions, overlapping or gapped spans, spans outside the canonical text, and tokens that do not reconstruct their source. Reject unknown kinds, altered token text, and an indexable rule other than text.

Reject duplicate query IDs, malformed query records, non-positive top_k or excerpt radius, missing ranking-rule version, and changed configuration. Check zero-term documents, an empty query, repeated query text, and an <unk> source token. Confirm inputs are not mutated.

Workspace

Keep identity, configuration, document, token, and query checks in this milestone. Do not construct postings, scores, excerpts, or output figures yet.

Hints

Review

Follow one accepted document from its identity through its token records. Then follow one rejected fixture to the exact rule that rejects it. What would be unsafe about silently applying a different tokenizer to a query?

How to check your work

Checks compare identities, spans, kinds, queries, and limits with the version-matched fixtures. The supplied fixture refuses incompatible inputs instead of repairing them.