Milestone 2 of 9

Select exact text tokens and preserve source positions

Retain selected-token and complete-token coordinates, exact spelling, source identity, and excluded-token counts without crossing documents.

Groups must be built from the same selected token stream in every document. Keep the searchable coordinate and the source coordinate together.

Goal

Apply the compatible TEXT-02 scanner, select only text records, retain exact token identity and both position systems, and account for excluded token kinds.

Inputs

Use validated token records and the accepted tokenizer and token-kind rule. Preserve exact spelling, case, non-ASCII characters, underscores, document order, and token order. Do not lowercase, stem, remove common words, join terms, or carry a sequence across a document boundary.

For each selected token retain stable document ID, zero-based position among selected text tokens, original full-token position, exact token text, and corpus, tokenizer, and source identity. Record excluded counts by kind so punctuation, spaces, and newlines remain visible in accounting.

Deliverables

Implement src/documents.py. Produce output/selected_tokens.jsonl with the selected records and both coordinates, plus excluded-token summaries by document and kind. Include zero selected tokens for empty and punctuation-only documents.

Checks

Check exact case, non-ASCII, underscore, punctuation, whitespace, repeated tokens, empty documents, and source-edge positions. Verify selected positions are contiguous and zero-based within each document, full-token positions are strictly increasing, token text and identities match the source, and selected plus excluded counts account for every scanner record.

Check no group can cross a document boundary, no ignored record becomes a selected term, and source token records remain unchanged. A different selection rule must be treated as a different analysis version.

Workspace

Keep token selection and excluded-kind accounting in src/documents.py. Write selected tokens and summaries only. Do not build windows or compare documents.

Hints

HintKeep two positions

Increment the selected-token position only for a text record; copy the scanner position separately. They answer different source questions.

HintCount what you ignore

Group construction should ignore punctuation as a token, but the report should not make it disappear. Count every excluded kind.

HintReset per document

Start selected positions at zero for each document. Never let the previous document's final position enter the next one.

Review

Choose a punctuation-separated pair of words. Which position sequence should a group use, and which should an excerpt use? What evidence proves punctuation was ignored but not lost?

How to check your work

Checks compare selected records, excluded counts, positions, and identities with the fixtures. The supplied fixture preserves exact token text and source coordinates under the supplied rule.