Milestone 5 of 8

Compare the systems item by item

Partition every aligned item into one of four paired outcomes and retain the IDs and labels behind each count.

Aggregate counts can hide which items changed. Compare the two systems for every aligned item and keep that partition available for inspection.

Goal

Classify each item into one paired-outcome category, save the item-level comparison, trace representative disagreements, and prove that the four categories conserve every item.

Inputs

Use output/aligned_items.csv with both match statuses. For each item, assign exactly one category:

  1. both_correct;
  2. baseline_only;
  3. candidate_only;
  4. both_wrong.

Keep item_id, expected label, both predictions, source group, and both match statuses in the comparison record.

Deliverables

Implement paired comparison in src/compare.py and save output/paired_comparison.csv plus a category-count artifact. Include one row per item and enough fields to explain how its category was assigned.

Select representative disagreements by item ID for the report. Do not call a candidate improvement causal or statistically meaningful; the comparison is a description of this supplied pair of prediction files.

Checks

Check that each item has exactly one category, the four category counts sum to the item count, and each category agrees with the two match statuses. Rebuild the counts from paired_comparison.csv and compare them with the saved category artifact.

Use a public fixture containing all four categories. Check that a shuffled aligned table produces the same item-level categories after sorting by ID and that expected or predicted labels are never replaced with a category name.

Workspace

Keep paired comparison in src/compare.py. Read aligned records from src/data.py and write the item-level and category artifacts under output/. Do not discard the item rows after calculating category counts.

Hints

HintFour categories cover four combinations
Think of each system's match status as correct or wrong. The four combinations are the complete partition.
HintKeep disagreements concrete
A representative disagreement is an item ID with its expected label and both predictions. It is not a reason inferred from the label.

Review

Choose one row from each category and verify the category from its two match statuses. Explain why item-level conservation is stronger evidence than a single difference between two overall fractions.

How to check your work

Checks compare the category schema and conservation evidence with the supplied fixture. The supplied fixture keeps aggregate counts traceable to IDs.