Milestone 7 of 9

Compare systems and create fair figures

Align per-query measures, calculate candidate-minus-baseline differences and unweighted macro means, and plot both systems with shared orders and scales.

The two systems should be compared on the same query/cutoff rows and displayed on the same visual scale. A difference on this fixture is an observation, not a general verdict.

Goal

Align per-query measures, calculate candidate-minus-baseline differences and unweighted macro means, and create comparable figures from exact plot data.

Inputs

Use retrieval_measures.csv, the validated query and cutoff display order, and the plot-order and scale configuration. For each query and cutoff, calculate candidate-minus-baseline differences for precision, recall, and F1. For each system and cutoff, calculate the unweighted arithmetic mean of the per-query measure, retaining query count and every contributing row. Do not pool denominators silently.

Deliverables

Implement src/compare.py and reporting in src/report.py. Produce output/paired_measure_differences.csv with aligned system values and candidate-minus-baseline differences, and output/macro_measure_summary.csv with system, cutoff, measure, unweighted mean, query count, and contributing rows.

Create comparable per-cutoff system figures and paired-difference figures. Save exact output/plot_data.json containing configured query/cutoff order, both system values, differences, macro values, labels, and shared scales.

Checks

Reject missing or duplicated query/cutoff/system rows before comparison. Check that every difference equals candidate minus baseline, every macro mean gives equal arithmetic weight to each contributing query, and query counts and rows are retained. A positive difference is an observation on this fixture, not proof of superiority or statistical significance.

Check both systems use identical query and cutoff order, measure axes use the shared [0, 1] scale, paired differences have a documented zero reference, labels are deterministic, and figures agree with plot_data.json. Do not add leaderboards, significance claims, pooled denominators, or alternative measures.

Workspace

Keep paired alignment in src/compare.py and figure/data generation in src/report.py. Read measure records without mutation. Write comparison tables, figures, and exact plot data under output/.

Hints

HintAlign before subtracting

Use (query_id, cutoff) as the key, then place baseline and candidate values side by side before calculating a difference.

HintMacro means are row means

Sum the per-query values and divide by the contributing query count. Do not reconstruct the mean from pooled hits or denominators.

HintShare the visual frame

Use the configured query and cutoff order and the same [0, 1] measure limits for both systems. Save those choices in plot data.

Review

Choose one query/cutoff and trace both measure rows into their difference. Then recompute one macro mean from its contributing rows. What would a pooled mean hide?

How to check your work

Checks compare differences, macro summaries, labels, scales, and plot data with the fixtures. The supplied fixture makes fair visual comparisons without claiming that a positive fixture difference generalizes.