Select flagged pairs and attach source evidence

A flagged list is a view over complete pair evidence. Sort it exactly, then join shared groups to every source occurrence before applying display limits.

Goal

Select flagged pairs under the primary rule, order them by exact similarity and generated ordinal, attach all shared-group occurrences, and render bounded source excerpts without discarding saved evidence.

Inputs

Use all_pair_metrics.csv, document group sets and occurrence records, complete selected tokens, full-token source records, and configured evidence limits. Do not call outputs confirmed duplicates; use flagged_pairs or candidates_under_rule.

Order flagged pairs by decreasing exact similarity fraction, then increasing generated pair ordinal for an exact tie. For each flagged pair, sort shared group tuples lexicographically under Python tuple/string order. Join every shared group to all occurrence records in both documents.

The configuration may limit rendered shared groups and occurrences, but saved complete group and occurrence rows must remain. For each rendered occurrence, select a bounded range of complete full-token records around its first and last full-token positions. Retain exact range, matched positions, and truncation metadata. Joined token text is display-only; truncation marks are not source content.

Deliverables

Implement selection and evidence in src/evidence.py. Produce ordered output/flagged_pairs.jsonl, complete output/shared_group_evidence.jsonl, and bounded output/review_excerpts.jsonl. Keep selected-token ranges, ordered full-token positions, group tuples, source identities, limits, and truncation flags in the records.

Checks

Check exact fraction ordering, generated-order ties, identical and partial matches, repeated shared groups, source-edge excerpts, overlapping excerpts, zero and non-zero evidence limits, punctuation and whitespace, non-ASCII and underscore terms, and occurrences with different selected/full-token ranges.

Verify every shared group joins all occurrences in both documents, complete evidence is not reduced by display limits, excerpt ranges clip at boundaries, and joined text does not alter source tokens. Do not select a document to keep or delete.

Workspace

Keep flagged ordering and shared evidence in src/evidence.py. Read complete pair, group, occurrence, and token artifacts without mutation. Write selected pairs, evidence, and excerpts; do not build sensitivity rows yet.

Hints

Review

Choose a flagged pair with a repeated shared group. How many occurrence rows should remain even if the rendered limit is one? Which order decides a tie?

How to check your work

Checks compare flagged order, shared groups, occurrence joins, excerpt ranges, and truncation metadata with the fixtures. The supplied fixture keeps complete evidence behind the bounded display.