Milestone 6 of 8

Build deterministic result excerpts

Locate the earliest matching full-token position, take the configured bounded token window, and record ranges, matches, and truncation without changing source text.

A ranked result should point back to source evidence. An excerpt is a bounded view of existing tokens, not a rewritten sentence.

Goal

For every returned result, select the earliest matched token position, keep a bounded token window around it, and record exactly which positions matched.

Inputs

Use ranked results, complete postings, source token records, query terms, and the configured excerpt radius. For a result, find the smallest full-token position belonging to any matched query term. Select at most the configured number of complete tokens before and after that position, clipped at document boundaries.

Deliverables

Implement excerpt selection in src/excerpts.py and save output/result_excerpts.jsonl. Each record should include query ID, document ID, earliest match position, selected half-open token-position range, token texts, and the positions that match query terms. Include explicit prefix and suffix truncation metadata when source tokens remain outside the selected range.

Join selected token text only for display. Do not inject HTML, alter source tokens, or use character offsets in place of token positions. A truncation marker is display metadata, not corpus content.

Checks

Use a match at the first token, a match at the last token, multiple matching positions, and a document with more context than the radius. Check that the earliest match wins, the range stays in bounds, selected positions are complete, and source records remain unchanged.

Run selection twice and compare complete JSON records. Check that a result with no positive score cannot receive an excerpt and that an excerpt never invents a source token. Check non-ASCII and underscore terms without changing their display text.

Workspace

Keep bounded window selection and display metadata in src/excerpts.py. Do not alter ranking or source records.

Hints

HintStore the range

The token texts make a useful preview, but the position range tells another program exactly which source records were shown. Save both.

HintEdges are ordinary cases

At the beginning or end of a document, fewer context tokens are available. Record that fact instead of padding the excerpt with invented text.

Review

Open one excerpt whose first matching term is not the first match encountered in a later, wider window. Can you identify the source positions without guessing from the joined display string?

How to check your work

Checks compare earliest positions, ranges, token lists, match positions, and edge flags with the fixtures. The excerpt makes lexical evidence easier to inspect; it does not make the result semantically relevant.