Milestone 4 of 8

Process queries with the same token rules

Normalize and scan each supplied query exactly like the corpus while retaining repeated text terms, ignored non-text tokens, missing terms, and explicit empty-result reasons.

A query must speak the same small language as the index. If the index keeps case and the query silently changes it, the result is not a fair comparison.

Goal

Normalize and scan each query with the versioned tokenizer, retain term order and repeated terms, and make ignored and missing terms visible.

Inputs

Use plans/queries.jsonl and the exact normalization and scanner configuration used for source documents. Select text tokens under the index rule. Preserve term text, case, underscores, non-ASCII characters, and the order in which terms occur. Count repeated query terms instead of collapsing them too early.

For every query, retain the original query text and analysis identity. Record punctuation, spaces, newlines, and other ignored token kinds separately.

Deliverables

Implement query processing in src/query.py and write output/query_analysis.jsonl. Each record should include query ID, original text, normalized text identity, ordered indexable terms, repeated-term counts, ignored-token records, present terms, missing terms, and an explicit outcome.

An empty query, an ignored-only query, or a query whose terms are all absent returns no result list with an explicit reason. This is different from a malformed query, which should be rejected.

Checks

Use empty, punctuation-only, whitespace-only, repeated, partially matched, fully matched, case-different, underscore, and non-ASCII queries. Confirm that query token positions remain available for diagnostics and that repeated terms remain countable. Check that missing terms are recorded even when another term is present.

Run the same query twice and compare its complete analysis record. Change the normalization or scanner version deliberately and check that the identity guard rejects it. Confirm source index and query inputs are not mutated.

Workspace

Keep tokenizer compatibility and query analysis in src/query.py. Do not calculate scores or rank documents in this milestone.

Hints

HintIgnored is not missing

A comma was scanned but is not an indexable term. A word absent from the index is a missing term. Record these cases separately.

HintKeep repetitions until scoring

The query red red blue has a different supplied score from red blue, so do not replace it with a set.

Review

Take one query containing a punctuation mark, a repeated term, and one absent term. Which parts should appear in the analysis record? Why would lowercasing the query create a different contract rather than a harmless convenience?

How to check your work

Checks compare ordered terms, counts, ignored records, missing records, and explicit empty outcomes with the fixtures. The query processor reports exact lexical behavior; it does not interpret intent.