Milestone 1 of 9

Verify corpus, tokenizer, index, queries, judgments, and configuration

Check source identities, complete query/document judgment coverage, token rules, formulas, system names, cutoffs, rounding, tolerances, and zero-term documents.

Numerical work is only meaningful when its inputs belong together. Begin by checking identities, complete judgment coverage, and every rule that will later affect a score or a rank.

Goal

Validate the corpus, tokenizer, positional index, query records, binary judgments, and retrieval configuration before building weights or rankings.

Inputs

Read accepted documents, complete scanner-token records, corpus and tokenizer identity, and the TEXT-04 positional index or version-matched fallback. Read plans/queries.jsonl, plans/relevance_judgments.csv, plans/retrieval_config.json, and the tiny fixtures.

Queries have unique non-empty IDs and exact query text under the TEXT-04 tokenizer and searchable-term rule. A judgment row has query_id, document_id, and relevant, where relevant is exactly integer 0 or 1, not a boolean-like string. Query/document pairs are unique and cover the complete Cartesian product of configured queries and corpus documents.

The configuration names exactly term_count_baseline and tfidf_cosine, formula and identity versions, natural logarithm, twelve-place score rounding, unique positive non-boolean integer cutoffs in display order, ranking ties, plot order, and numerical tolerances.

Deliverables

Implement the compatibility checks in src/config.py and clearly separated loaders. Produce an ordered validated input record containing source identities, document and query order, token-rule identity, judgment coverage, system names, formula versions, cutoffs, tie rules, rounding, plot order, and tolerances.

Checks

Reject changed corpus, tokenizer, or index identity; altered token selection or normalization; missing, duplicate, extra, or unknown document and query IDs; changed manifest or query order; malformed query records; and incomplete, duplicate, extra, or unknown judgment pairs. Reject any judgment other than exact 0 or 1.

Reject unknown systems or formula versions, a non-natural-log rule, invalid rounding, non-positive or boolean-like cutoffs, duplicate cutoffs, invalid tie or plot order, and invalid tolerances. Check empty and zero-term documents, repeated, shared, case-different, non-ASCII, underscore, ignored, and missing query terms. Confirm inputs are not mutated.

Workspace

Keep identity, configuration, and input-coverage checks in src/config.py and its loaders. Do not calculate frequencies, weights, scores, measures, plots, or reports yet.

Hints

HintCheck the Cartesian product

Build the expected query/document key set from the validated manifests, then compare it with the judgment keys. Row count alone cannot prove complete coverage.

HintA zero is a judgment

The integer 0 is valid evidence. Do not coerce strings such as "0", false, or another integer into the contract.

HintFreeze the rules

Carry every version, order, cutoff, and tolerance into the validated configuration so later modules do not invent defaults.

Review

Follow one query/document pair from the manifests into its judgment row. Then inspect a broken fixture with one missing or duplicate pair. Why would a plausible measure be unsafe if the judgment table did not cover every pair?

How to check your work

Checks compare identities, coverage, systems, formulas, cutoffs, and tolerances with the supplied fixtures. The supplied fixture rejects incompatible or incomplete inputs rather than repairing them.