Milestone 5 of 9

Compare every eligible document pair exactly once

Generate every manifest-ordered unordered pair, retain eligible and excluded rows, calculate exact metrics, and prove complete pair accounting.

Now compare the whole bounded corpus. Exhaustive does not mean unordered duplicates: each pair gets one stable identity and remains in accounting.

Goal

Generate every unordered pair of distinct documents exactly once in corpus-manifest order, retain eligible and excluded records, calculate exact metrics and decisions, and prove pair-count and no-reversal invariants.

Inputs

Use validated document groups and primary setting. For n documents, generate exactly n * (n - 1) / 2 rows with the earlier manifest document first. The supplied document bound makes exhaustive quadratic comparison intentional and inspectable; do not imply unbounded scale.

For every pair retain generated pair ordinal, document IDs, eligibility and exact exclusion reason, and, when eligible, intersection count, union count, exact fraction, display decimal, threshold products, and Boolean decision. Every eligible pair, including an unflagged zero-overlap pair, remains in the complete table.

Deliverables

Implement pair generation and comparison in src/pairs.py. Produce output/all_pair_metrics.csv with every pair in generated order, eligible and excluded records, exact numerators and denominators, threshold products, decisions, and primary-setting identity.

Checks

Check one, several, and short or empty documents; exact copies, insertion, removal, reordered passages, repeated groups, unrelated text, partial overlap, and zero overlap. Verify the complete row count, no reversed duplicate, earlier document first, consecutive generated ordinals, and eligible plus excluded counts equal the total.

Check pair metrics agree with the hand-calculated sets, excluded pairs retain reasons and no decision, eligible zero-overlap pairs remain, and primary threshold equality is included. Confirm every input is immutable.

Workspace

Keep exhaustive pair generation, ordering, eligibility, and metrics in src/pairs.py. Write the complete pair table only. Do not filter to flagged pairs or attach excerpts yet.

Hints

HintUse two indices

For each document index i, pair it only with later indices j. This creates one manifest-ordered orientation without a second deduplication pass.

HintCount before filtering

Compute the expected quadratic row count first. A selected list cannot reveal whether an excluded pair was lost.

HintKeep excluded rows

Eligibility is part of the result. Store a reason and leave similarity and threshold fields absent rather than inventing zero overlap.

Review

List all pairs for four documents and label their generated ordinals. Which pair would a reversed-orientation bug duplicate, and how would the invariant expose it?

How to check your work

Checks compare the complete table, ordinals, orientations, counts, and reasons with the fixtures. The supplied fixture is exhaustive for the bounded corpus and never calls the selected view the complete evidence.