Compare runs from saved evidence

Comparison starts with the question, “What changed?” before it asks, “What result changed?” Read the saved evidence rather than comparing live objects.

Goal

Choose one baseline run explicitly, compare configurations before common result fields, handle failed runs honestly, and save comparison.csv and comparison.png.

Inputs

Use the completed and failed run directories and experiment_manifest.json from the previous milestone. The reader chooses the baseline by safe run name; the code must not silently choose the first or best-looking result.

Compare the fields seed, scale, offset, and noise_bound first. Then compare common result summaries and values. A failed run has unavailable result values, not zeros. Floating-point differences need an exact representation or an explicit tolerance rule and the recorded runtime.

Deliverables

Implement src/compare.py and write:

• comparison.csv with baseline, candidate, changed settings, status, common result fields, differences, and unavailable values;
• comparison.png generated from saved run evidence, with a clear legend and run names; and
• a short comparison section in report.md naming the baseline and the one setting deliberately changed.

The table should expose setting differences before result differences. It may describe a difference, but it must not call one run better without a supplied criterion.

Checks

Use a same-seed pair, a pair with one changed setting, and the failed run. Check that the baseline is explicit, configuration values are read from config.json, and result fields are read from saved result or values files. Check exact fields and tolerance-aware numerical fields separately.

Mutate a saved result after the run and confirm that the comparison reports the changed evidence rather than silently using an in-memory copy. Check that the failed row preserves its status and unavailable values. The plot must use saved values and identify its runs; it must not be a decorative plot of a new calculation.

Workspace

Keep comparison logic in src/compare.py. Write comparison artifacts at the project root and keep run directories unchanged. src/main.py may call the comparison after the manifest is written, but it should pass an explicit baseline.

Hints

Review

Read the table from left to right: baseline, candidate, changed settings, status, then results. Confirm that a reader can tell what was changed before seeing the numerical difference, and that the figure can be traced to saved values.

How to check your work

Checks compare the table schema and failure treatment with the supplied fixture. The supplied fixture shows one comparison format; it does not rank the supplied configurations.