Milestone 7 of 8

Replay one completed run

Reconstruct one run in a separate directory and report exact or tolerance-aware agreement at the first differing field.

Replaying a run means rebuilding its evidence from its saved inputs. It must not overwrite the original directory or quietly widen the claim beyond the runtime that was recorded.

Goal

Load one completed run's configuration and source identity, execute it in a separate replay directory, and record whether values, summaries, figure data, and provenance agree.

Inputs

Use one completed run under runs/<run_name>/, its config.json, the original source identity, and the runtime and generator evidence in the manifest. The replay must use a separate directory such as replays/<run_name>/; never write over the original run.

The replay boundary includes the compatible Python, NumPy, and Matplotlib versions and the generator implementation. If the runtime is incompatible, record that fact instead of claiming an exact replay.

Deliverables

Add replay logic to src/runner.py or a clearly separated function and write:

  • a separate replay directory with the same inspectable output artifacts;
  • a replay agreement or mismatch record;
  • the first differing artifact or field when the replay does not agree; and
  • a replay section in report.md naming the source run, replay directory, runtime boundary, and conclusion.

The replay record must compare source identity, configuration, summaries, values, and figure data. It should distinguish a matching computation from a matching file byte stream when the image encoder or metadata may differ.

Checks

Replay an unchanged completed run and check agreement for the source identity, configuration, values, summaries, and plot data. Confirm that the original files' hashes or contents remain unchanged and that the replay directory is distinct.

Deliberately change one saved input or use an incompatible runtime fixture and check that the mismatch record names the first differing artifact or field. Do not reduce every mismatch to “replay failed.”

Workspace

Keep replay output under replays/ and preserve runs/ as original evidence. Use the root manifest for runtime and source identity. Keep the comparison artifacts separate from replay output so a later comparison cannot overwrite a replay record.

Hints

HintReplay from saved evidence
Do not use the current editor state as the replay input. Load the configuration and source identity from the completed run.
HintName the first disagreement
Compare the source and configuration before values. If those agree, compare summaries and values before comparing figure metadata.

Review

Explain what “reproducible” means in this project and where the claim stops. Check that the replay has not changed the original evidence and that the report does not promise agreement under every future dependency version.

How to check your work

Checks compare the replay record and mismatch vocabulary with the supplied fixture. The supplied fixture keeps the original run immutable and makes the runtime boundary explicit.