Milestone 3 of 8

Analyze phrase and proximity queries

Apply the compatible scanner, retain selected and ignored tokens, validate exact arity, and record invalid, missing, repeated, case-different, and non-ASCII cases.

Before matching, turn each query into a record that shows what the tokenizer kept and what it ignored. Invalid input should be visible as invalid, not silently converted into a different query.

Goal

Analyze every phrase and proximity query with the corpus tokenizer and record complete tokens, selected searchable terms, ignored tokens, validation status, and missing-term evidence.

Inputs

Use the validated query records and the same versioned normalization and scanner as the corpus. A phrase's text must produce at least two searchable terms. A proximity query's left and right fields must each produce exactly one searchable term. Its maximum_intervening_terms is an integer greater than or equal to zero, and its direction is exactly after or either.

Preserve exact token text, case, non-ASCII characters, and underscores. Record all complete scanner tokens, selected text terms, and ignored punctuation, spaces, and other token kinds. A valid query whose term is absent from the index remains valid but carries an explicit missing-term reason.

Deliverables

Implement src/queries.py. Produce one deterministic JSONL analysis record per input query in query-file order, including query ID, original structured fields, complete tokens, selected terms, ignored tokens, validation status, normalized term values under the supplied rule, missing terms, and a rejection reason when applicable.

Checks

Check valid two-term and longer phrases, repeated terms such as a a, punctuation and whitespace, case-different terms, non-ASCII terms, underscores, missing terms, and ignored-only fields. Reject empty or one-term phrases, multi-token proximity sides, missing fields, duplicate IDs, unsupported kinds or directions, and malformed or negative limits.

Ensure invalid records do not produce matches. Ensure missing terms produce no matches later while retaining the missing-term evidence. Preserve query-file order and verify that analysis does not mutate the input JSONL or tokenizer records.

Workspace

Keep scanning and validation in src/queries.py; keep matching out of this milestone. Save output/query_analysis.jsonl with deterministic key and record order suitable for later read-back.

Hints

HintAnalyze fields separately

A phrase has one text field. A proximity query has two term fields. Keep each field's complete and selected tokens so a rejected side can be inspected.

HintMissing is not invalid

A valid term that is absent from the index is still a valid query. Separate its missing-term evidence from schema errors.

HintUse the published tokenizer

Do not lower-case or strip terms because that makes a query appear to match a differently built index.

Review

Inspect one valid query, one missing-term query, and one rejected query. Can a reader tell whether no result means “the term is absent” or “the query was invalid” without rerunning the tokenizer?

How to check your work

Checks compare analysis records with the version-matched scanner fixtures. The reference keeps complete, selected, and ignored tokens visible and rejects ambiguous arity instead of guessing the reader's intent.