Project 3
Build an Experiment Runner
Run one supplied computation under several explicit configurations, keeping each run's settings, results, status, and files together for comparison and replay.
Project question: Can you run several explicit configurations, keep each result separate, and show which evidence can be repeated?
What makes this a project
An experiment is more than a number printed at the end of a program. You need to know which settings produced it, which source data were used, whether the run completed, and whether another run agrees with it. In this project you will build a small runner around a supplied computation. The computation is read-only; your work is the plan, the execution boundary, and the evidence.
You will build a system that can:
- validate an ordered plan before running anything;
- keep every run in its own directory;
- record completed and failed runs without inventing missing results;
- compare settings before comparing result fields;
- replay a completed run in a separate directory; and
- explain one same-setting replay and one controlled change in a short report.
The system is deliberately small. It is not a service, dashboard, scheduler, or general experiment-tracking product. The point is to practise the habits that make a numerical result inspectable.
What you should know first
The project follows the Python path through Chapter 13. You should be able to use functions and records, organize a multi-file program, validate inputs, work with arrays and plots, save JSON and CSV files, and use controlled randomness. DATA-02 is useful preparation because it introduces manifests, replay, and comparison, but it is optional. The supplied files provide the same boundary if you did not complete it.
The supplied computation
The read-only src/experiment.py exposes:
It applies the visible scale and offset settings to supplied measurement
data, adds bounded variation using seed and noise_bound, calculates named
summaries, and returns values needed for one plot. Read the implementation and
hand-check a tiny input, but do not rewrite it. The assessed work is how you
run and preserve the computation.
Run configuration
Each configuration has these fields:
| Field | Rule |
|---|---|
run_name | One to 40 lowercase ASCII letters, digits, and internal hyphens; it starts and ends with a letter or digit |
seed | Integer from 0 through 4_294_967_295 |
scale | Finite number in [-10.0, 10.0] |
offset | Finite number in [-100.0, 100.0] |
noise_bound | Finite number in [0.0, 10.0] |
The run plan is ordered. Reject duplicate or unsafe names and invalid numbers
before starting any run. A run directory must be formed from a validated name;
never accept an absolute path, .., or another path component from a plan.
Evidence boundary
Each run owns one directory:
runs/<run_name>/
config.json
result.json
values.csv
figure.png
run_status.json
Write run_status.json as completed only after every required artifact has
been written and read back successfully. The supplied failure adapter uses the
same interface and raises one documented exception. A failed run keeps its
configuration, status, stable failure stage, and short error message; it does
not receive plausible result files or a completed status. Never silently
overwrite an existing run directory.
The root experiment_manifest.json records the source dataset identity,
experiment interface revision, ordered run names, configuration and status
paths, relevant Python/NumPy/Matplotlib versions, and comparison artifacts. It
references run files instead of copying every result into one large record.
Project milestones
Work through these in order. Each milestone produces evidence used by the next one, while project completion remains separate from lesson progress.
- 1Inspect the experiment interface and one expected run
Trace a tiny input through the supplied computation and identify the configuration, values, summaries, and required files.
- 2Define and validate a run plan
Validate ordered configurations, safe unique run names, and numerical bounds before any run directory is created.
- 3Execute and save one complete run
Write one isolated run, read every required artifact back, and record completion only after all checks succeed.
- 4Record one failed run honestly
Use the supplied failure adapter to preserve configuration and stable failure evidence without inventing unavailable results.
- 5Execute a small run plan
Run independent configurations in ordered directories and build a root manifest that references their saved evidence.
- 6Compare runs from saved evidence
Choose a baseline, show configuration differences first, and compare completed and failed results without ranking them.
- 7Replay one completed run
Reconstruct one run in a separate directory and report exact or tolerance-aware agreement at the first differing field.
- 8Write the experiment report
Explain one replay, one setting change, one failure, and the limits of the saved evidence in a short report.
How completion works
Project progress is separate from chapter and lesson progress. A milestone is ready to mark done when its implementation, saved evidence, and checks are complete. Passing an automated check does not prove that your report explains the evidence, so inspect the files and complete the review as well.
The reusable boundary is the adapter, configuration and result records, safe run identity, run-directory layout, status vocabulary, manifest, comparison interface, and replay record. Later projects may replace the supplied computation while keeping this boundary. They must provide a compatible version-matched fixture if this project was skipped.
Limits
This project does not introduce arbitrary plugin discovery, shell commands, concurrency, scheduling, queues, remote workers, databases, dashboards, authentication, resumable partial computation, automatic retries, hyperparameter optimization, experiment-tracking services, or causal claims. The replay claim is limited to the recorded compatible runtime and generator implementation; it is not a promise about every future library version.
Review
The final review asks whether invalid plans are refused before execution, completed and failed runs are represented honestly, existing evidence is protected, configuration is compared before results, and replay happens in a new directory. The report should name what the evidence supports and what it does not support. A run is not a model, and one setting is not universally best merely because its output looks attractive.