Build an Evaluation Report

Project question: Can you align two prediction files with the expected labels, calculate the supplied count-based measures, and explain where the two systems agree or differ without claiming more than the records show?

What makes this a project

An evaluation starts with identity, not with a score. In this project you will join expected labels and two sets of predictions by item_id, calculate overall and per-label counts, inspect neutral source-group slices, compare the systems item by item, and save plots and a report that can be replayed.

The project implements and audits supplied definitions. Later subjects explain what those measures can establish. A match fraction on this small fixture is not proof of future performance, generalization, or a reason to rank systems without a stated question.

You will produce:

• one aligned record for every item;
• overall summaries and expected-row/predicted-column confusion tables;
• per-label results with explicit undefined ratios;
• descriptive source-group slices with visible denominators;
• a four-category paired comparison of the two systems;
• fair numerical and visual summaries;
• saved artifacts and a separate replay record; and
• a concise report whose claims point to item-level or aggregate evidence.

What you should know first

The project follows the Python path through Chapter 13. You should be able to validate and join records by stable IDs, count and group with dictionaries, construct arrays and tables, handle zero denominators explicitly, create comparable plots, save manifests, and replay a deterministic report. NUM-02 and DATA-03 are useful preparation but optional. The supplied records and formulas provide the local boundary if you did not complete either project.

Supplied records and configuration

Use these read-only inputs:

• data/items.csv contains unique item_id, expected_label, and a neutral non-empty source_group used only for descriptive slices;
• data/baseline_predictions.csv contains one unique item_id and predicted_label per item;
• data/candidate_predictions.csv contains the same required item IDs and one prediction each, deliberately stored in a different row order;
• plans/evaluation_config.json gives the ordered allowed labels, display names, and slice field.

Every expected and predicted label must belong to the configured label order. The prediction files must cover exactly the item IDs in items.csv; row position is never an alignment key. The neutral source groups are fixture names, not demographic categories or a fairness claim.

Supplied measures

For each system, calculate:

correct_count = number of items where predicted_label == expected_label
total_count = number of matched items
match_fraction = correct_count / total_count

The project calls this a match fraction for the supplied records.

Build a square count table in configured label order. The row is the expected label, the column is the predicted label, and the cell is the number of items with that pair. The table total equals the item count; its row sums equal expected-label counts and its column sums equal predicted-label counts.

For each label, save expected count, predicted count, diagonal correct count, recall = correct_count / expected_count when the denominator is non-zero, and precision = correct_count / predicted_count when the denominator is non-zero. Save an undefined ratio as JSON null and display it as not defined; never replace it with zero.

For each source_group, save item count, correct count, and match fraction for each system. Always show the denominator. Do not rank or explain group differences from this small fixture.

For each item, classify the two outcomes into exactly one category:

1. both_correct;
2. baseline_only;
3. candidate_only;
4. both_wrong.

The four category counts must sum to the item count, and the item-level file must preserve the IDs and labels behind every aggregate.

The project workspace

project/
  README.md
  data/items.csv                    # supplied, read only
  data/baseline_predictions.csv     # supplied, read only
  data/candidate_predictions.csv    # supplied, read only
  plans/evaluation_config.json      # supplied starting config
  src/config.py                     # reader implementation
  src/data.py                       # reader implementation
  src/counts.py                     # reader implementation
  src/compare.py                    # reader implementation
  src/report.py                     # reader implementation
  src/main.py                       # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py             # supplied, read only

Keep identity matching, count construction, paired comparison, and reporting independently inspectable. Maintain one continuous workspace through every milestone.

Required evidence

The completed project contains:

• aligned_items.csv with expected, baseline, candidate, source group, and both match statuses;
• overall summaries for both systems;
• baseline_confusion.csv and candidate_confusion.csv;
• per_label_results.csv and slice_results.csv;
• paired_comparison.csv plus category counts;
• confusion, paired-outcome, and slice figures;
• evaluation_manifest.json and a replay agreement or mismatch record; and
• a concise project report at report.md.

Checks should include shuffled input order, duplicate/missing/extra and unknown-label records, hand-built confusion tables, total/row/column invariants, labels with no expected or predicted items, exact null handling, slice denominators, four-way conservation, traceability to item IDs, shared plot order and scales, no input mutation, artifact read-back, and replay. Automated checks cannot decide whether an explanation is justified; the final review must do that.

Limits

This project does not teach statistical significance, confidence intervals, calibration, probability scores, thresholds, ROC/AUC, ranking measures, regression measures, class-imbalance policy, fairness evaluation, causal claims, hidden test sets, model selection, deployment monitoring, or future performance. The supplied names and labels are enough to test the mechanics; they do not justify a broader claim about a system.

Review

The final review asks whether every item is aligned by ID, every measure is derived from the same records, undefined ratios remain visible, slices retain denominators, paired categories conserve all items, plots use shared orders and scales, replay uses a separate directory, and the report distinguishes observation from interpretation.