Build and hand-check the two-coordinate positional view

The engine needs two answers to “where is this term?” One position answers the phrase question; the other points back to the complete source token stream.

Goal

Derive one searchable-term stream per document while retaining each occurrence's zero-based searchable position and zero-based full-token position. Prove the new view agrees with the accepted source and TEXT-04 index.

Inputs

Use the validated documents and complete scanner-token records from Milestone 1. Select only text records, in complete scanner order. For each occurrence, retain exact token text, document ID, corpus identity, tokenizer identity, indexed_position, and full_token_position.

The indexed_position values are contiguous from zero within each document. The full_token_position values are positions in the complete scanner output; punctuation, spaces, and newlines remain visible there but are not searchable. Store terms in lexicographic text order and postings in corpus-manifest document order. A document with no searchable terms stays in the accounting with an empty stream.

Deliverables

Implement src/positions.py and produce output/positional_term_index.json with both coordinates on every occurrence. Produce hand-check records for a repeated term, punctuation and whitespace, a non-ASCII token, an underscore, and an empty document.

Checks

For every document, check that searchable positions are exactly 0, 1, ..., n - 1, full-token positions strictly increase, and each occurrence has the expected document, term, and identity. For every term, check lexicographic term order, manifest posting order, and occurrence order.

Check that searchable counts equal the selected text records, full-token positions and counts agree exactly with TEXT-04, and total occurrences equal the sum of document searchable lengths. Check that source token records and the supplied index remain unchanged.

Use a tiny example in which punctuation separates two terms: the terms may be adjacent in indexed_position while their full_token_position values are not. Do not accidentally make punctuation searchable or compress source positions.

Workspace

Keep derivation and conservation checks in src/positions.py. Write only the positional-index artifact and hand-check evidence. Do not analyze queries or construct matches and excerpts yet.

Hints

Review

Choose one repeated term and write both coordinate sequences by hand. Which sequence should phrase matching use? Which sequence should an excerpt use, and why would confusing them hide punctuation from the source evidence?

How to check your work

Checks compare the tiny positional index with the scanner records and TEXT-04 posting fixture. The supplied fixture preserves every occurrence and adds only the contiguous searchable coordinate; it does not alter token text or source positions.