Save and replay the evaluation

An evaluation is easier to inspect when its alignment, definitions, tables, plots, and runtime can be loaded again. Save the evidence before writing the conclusion.

Goal

Write aligned items, all summaries and tables, plots, configuration, source identity, runtime evidence, and a manifest. Replay the evaluation in a separate directory and record agreement or the first mismatch.

Inputs

Use all artifacts from the earlier milestones and the original files under data/. The manifest must identify the dataset and schema, configuration and label order, source row identity, generated artifact paths, relevant Python/NumPy/Matplotlib versions, and implementation revision.

The replay starts from saved inputs and configuration and writes to a separate directory such as output/replay/. Never overwrite the original aligned table, summaries, comparison, or figures.

Deliverables

Extend src/main.py and src/report.py, or add a clearly separated replay function, to save:

• output/aligned_items.csv;
• overall summaries;
• baseline and candidate confusion tables;
• output/per_label_results.csv and output/slice_results.csv;
• output/paired_comparison.csv and category counts;
• confusion, paired-outcome, and slice figures;
• output/evaluation_manifest.json; and
• a replay agreement or mismatch record under output/replay/.

The manifest should reference artifacts rather than copying every table into one large record. The replay record compares item IDs, labels, match statuses, counts, ratios including null, slices, paired categories, and plot data.

Checks

Read every required artifact back and check its schema, identity, counts, label order, and links. Replay an unchanged evaluation in a separate directory and compare aligned records, tables, summaries, category counts, and plot data. Confirm that the original files remain unchanged.

Deliberately change one saved prediction, label order, or aggregate artifact and check that the replay record names the first differing field or file. If the runtime is incompatible, record that limitation instead of claiming exact agreement.

Workspace

Keep original artifacts under output/ and replay artifacts under a distinct child directory. Keep output/evaluation_manifest.json and the project-level report.md at stable paths throughout the project.

Hints

Review

Open the original and replay directories side by side. Explain what agreement establishes for these supplied records and why it does not establish future performance, significance, fairness, or a causal improvement.

How to check your work

Checks compare the manifest and replay record with the supplied fixture. The supplied fixture keeps original evidence immutable and makes a mismatch inspectable.