Milestone 3 of 9

Build unique groups and complete occurrences

Construct every overlapping exact tuple, keep all occurrence coordinates, and classify documents that cannot form a group.

An overlapping group has an exact tuple identity and a source occurrence. Keep both the set used for similarity and every occurrence used for evidence.

Goal

Build all fixed-length overlapping group tuples, retain complete occurrences, and classify empty and short documents without crossing document boundaries.

Inputs

Use selected tokens and the primary positive group size k. Start at every selected-token position with k complete consecutive selected tokens. Advance one selected token at a time. The group value is the exact ordered tuple of token strings; never join tokens into an ambiguous display string.

For each document retain the set of unique group tuples for similarity and an occurrence record for every window with selected-token half-open range [start, start + k), ordered full-token positions, exact tuple, and stable source identity. Repeated occurrences collapse only in the set. A document with fewer than k selected tokens has an empty set and status insufficient_groups.

Deliverables

Implement src/groups.py. Produce output/document_groups.jsonl with unique group tuples, selected-token counts, and status, and output/group_occurrences.jsonl with every occurrence and both coordinate systems. Include repeated groups rather than deduplicating the occurrence table.

Checks

Hand-check a document with exactly k tokens, a longer document with overlapping windows, repeated groups, an empty document, and a short document. Verify the number of windows is max(0, selected_count - k + 1), tuples retain order and exact text, selected ranges are half-open and contiguous, full-token positions align with the source, and no window crosses a document boundary.

Check unique-set size against occurrence count, preserve duplicate occurrences, and confirm status insufficient_groups is not treated as a duplicate or a missing document. Do not use a joined display string as group identity.

Workspace

Keep window creation, tuple identity, set construction, and occurrence records in src/groups.py. Read selected tokens without mutation. Do not generate pairs or threshold decisions yet.

Hints

HintAdvance one token

For selected tokens a b c d and k = 3, the windows begin at positions 0 and 1. Overlap is part of the contract.

HintSet and table have different jobs

Use a set for one document's similarity groups. Keep a list or table for every occurrence so repeated source evidence survives.

HintUse ranges, not joined strings

Store [start, start + k) and the ordered source positions. This keeps repeated and punctuation-separated groups unambiguous.

Review

Write the windows for a b a b at k = 2. Which tuple repeats, how many occurrence rows remain, and how many unique set members should similarity use?

How to check your work

Checks compare group tuples, unique sets, occurrences, ranges, and short-document statuses with the hand-check fixtures. The supplied fixture deduplicates only the similarity set, never the source evidence.