Milestone 5 of 8

Execute a small run plan

Run independent configurations in ordered directories and build a root manifest that references their saved evidence.

One run is useful only when the runner can repeat the same boundary for a small ordered plan. Keep runs independent and gather their evidence in one manifest.

Goal

Execute at least three valid configurations, including one completed and one failed run, without sharing mutable results or accidental generator state, then write experiment_manifest.json.

Inputs

Use the validated plan from the previous milestone, the supplied readings and dataset manifest, and the normal computation plus failure adapter. The plan must have at least three distinct safe names. Include a same-seed configuration when you want to compare reproducibility, and include one deliberate setting change for the later comparison.

The manifest must record source dataset identity, experiment interface revision, ordered run names, configuration and status paths, relevant Python, NumPy, and Matplotlib versions, and the paths of later comparison artifacts.

Deliverables

Implement the plan loop in src/main.py or a small orchestration function. It must validate the complete plan before the first execution, run each configuration independently, preserve every run directory, and continue to record a supplied computation failure without making it look successful.

Write experiment_manifest.json only after the ordered run inventory and runtime evidence are available. It should reference run files rather than copying all results into one large JSON object. Record the plan revision and the runner or adapter revision used for each run.

Checks

Check that the plan order is preserved in the manifest, each run name appears once, and every referenced configuration and status path exists or is recorded as unavailable for the documented failure. Check that a failed run remains failed and that a later run cannot reuse its mutable result object.

Run a plan with two equal configurations under separate names. Their saved configuration, values, and summaries should agree under the recorded runtime; their directories and identities must remain different. Check that the runner does not reuse a global random generator or results list between runs.

Workspace

Use src/main.py for plan orchestration and src/runner.py for one-run execution. Keep all generated run directories under runs/ and the root manifest at experiment_manifest.json. Keep the supplied plan and source files read-only.

Hints

HintValidate once, execute many
The plan boundary comes before the loop. Do not discover an invalid name after earlier runs have already produced evidence.
HintGive each run fresh state
A new configuration and run directory are not enough if a result list or generator lives outside the loop. Make the state passed to one run explicit.

Review

Read the manifest and follow each path to its run directory. Explain why a failed run belongs in the ordered evidence even though it has no completed result. Check that equal configurations are reproducible without being merged or overwritten.

How to check your work

Checks compare the manifest and orchestration order with the supplied fixture. The supplied fixture records provenance without turning the manifest into a second copy of every artifact.