Milestone 3 of 9

Analyze queries and build sparse query vectors

Apply compatible token rules, preserve repeated, ignored, and missing terms, reuse corpus idf, and represent zero-norm queries explicitly.

The two systems must receive the same query evidence. Analyze query text with the corpus rules, then turn only in-index terms into a sparse weighted vector.

Goal

Apply compatible tokenizer rules to every query, preserve repeated, ignored, and missing terms, and calculate sparse query weights and norms using the corpus idf values.

Inputs

Use validated query records, the same versioned normalization and scanner as TEXT-04, the indexed term statistics, and the supplied query order. For every term present in the index:

query_weight(t, q) = query_term_count(t, q) * idf(t)
query_norm(q) = sqrt(sum(query_weight(t, q) ** 2 for t in in-index query terms))

Missing terms remain in query evidence but contribute no vector component. A query with no in-index terms has norm zero and produces an explicit empty ranking for both systems. Do not invent an idf or document contribution for a missing term.

Deliverables

Implement src/queries.py. Produce one deterministic output/query_analysis.jsonl record per query with complete scanner tokens, selected terms, ignored terms, repeated counts, missing terms, and query identity. Produce output/query_weights.jsonl with in-index term counts, idf, weights, query norm, and an explicit zero-norm status.

Checks

Check empty, one-term, repeated, shared, case-different, non-ASCII, underscore, ignored-only, and missing-term queries. Verify the selected terms use exactly the corpus token rule and query order. Check every saved weight uses the matching corpus idf, every norm agrees with its weight squares, and missing terms have no invented vector component.

Check zero-norm queries create no candidates or ranked documents for either system while retaining their analysis and explicit empty-ranking reason. Check query-file order, deterministic JSONL, and input immutability.

Workspace

Keep scanning and sparse query-vector construction in src/queries.py. Read the validated term statistics without changing them. Do not score documents or join judgments yet.

Hints

HintCount before weighting

Repeated query terms affect query_term_count; ignored and missing terms do not become vector entries. Keep all three categories visible in analysis.

HintReuse corpus idf

Look up each in-index query term in the saved term statistics. A missing lookup is evidence of absence, not a reason to make up a new idf.

HintMake zero explicit

Store query norm zero and an empty-ranking reason. Avoid a division attempt later by handling this state at the query boundary.

Review

Trace a repeated query term through its scanner records, count, idf lookup, weight, and norm. Then inspect a missing-term query. Can the two systems be compared fairly if one silently uses a different token rule?

How to check your work

Checks compare analysis and query-weight records with the supplied fixtures. The reference reuses corpus idf, preserves missing evidence, and gives zero-norm queries an explicit empty ranking.