Rerun and Compare

Reconstruct one saved run, define exact and tolerance-aware agreement, locate a first mismatch, and distinguish replay from a one-setting comparison.

A saved run record contains settings and measurements. Reproduction means loading that evidence, reconstructing the run from its settings, and checking that the new output agrees under a stated rule. It is different from running similar code twice or looking at two plots that appear close.

We will replay bounded noise added to the calibrated measurement table:

Reconstruct the Configuration First

Read the JSON record written by the previous lesson, then reconstruct its frozen configuration:

This is the local-file workflow. A standalone browser run can receive the same JSON document as a supplied input fixture; it should not depend on a temporary file left by another page.

Check configuration before output. If the new run used a different bound or sample count, an output mismatch would be expected rather than mysterious:

assert replay_config == saved_config

When configurations differ, report the first differing field:

This check prevents us from debugging numerical output before confirming that the two runs were asked to do the same work.

Q1. Check settings before generated values

A replay uses the saved seed and sample count but changes noise_high from 0.2 to 0.1. What should the comparison report first?

Choose one

Report the noise_high configuration mismatch before comparing generated arrays.

Ignore configuration and inspect only the first noisy value that differs.

Call it an exact reproduction because the seed is unchanged.

Select one choice, then check.

HintCompare causes before effects

The noise bound controls the generated values. Establish whether that input matches before diagnosing the output.

SolutionThe noise bound differs first

Report that noise_high changed from 0.2 to 0.1. The new output belongs to a controlled comparison, not an exact replay.

Compare Discrete and Numerical Results Differently

Define one complete run from the configuration:

Rebuild the saved result from the loaded JSON values. The new result comes from executing the experiment once with the reconstructed configuration:

Observation identifiers are discrete. They should match exactly and in the same order:

Noisy readings and means are floating-point results. Compare them with np.allclose:

Within this controlled program, the same configuration, generator algorithm, and call order reproduce the same values. A seed alone is not a promise of exact replay across changed software, algorithms, or execution paths. A fuller system would record that environment information; this lesson keeps the boundary to one unchanged program.

Q2. Choose exact and tolerance-aware checks

Which checks match the two kinds of result?

Choose one

Use np.array_equal for ordered observation IDs and np.allclose for noisy floating-point readings.

Use tolerance-aware comparison for observation IDs because their order may change.

Skip numerical checks when the replay plot looks similar.

Select one choice, then check.

HintMatch the check to the value's role

Observation IDs identify exact records. Generated measurements are numerical values produced through floating-point computation.

SolutionUse exact identity and numerical closeness

Use np.array_equal for the ordered IDs. Use np.allclose for noisy readings and their floating-point summaries.

Locate the First Output Mismatch

If configurations agree but numerical values do not, locate the first divergence instead of inspecting the whole table at once:

Trace that position backward through the call order: selected calibrated value, generated noise value, generator state before the draw, and configuration. The first differing stage narrows the repair. Do not change a tolerance merely to hide a disagreement whose cause has not been found.

Separate Replay from a Controlled Comparison

A controlled comparison deliberately changes one setting while holding the others fixed. replace creates that configuration visibly:

comparison_config = replace(saved_config, noise_high=0.1)

Verify that exactly one field changed:

Then run and compare:

same_values is expected to be False. That does not mean reproduction failed; this branch asked a different question. Report the changed field beside the output difference so the comparison remains interpretable.

Q3. Replay, then change exactly one setting

Complete the replay and controlled-comparison checks. The replay must match; the comparison must change only noise_high and produce different values.

Editable Python

HintKeep replay and comparison contracts separate

Use replay_config == saved_config, np.array_equal(replay_ids, saved_ids), and np.allclose(replay_values, saved_values). Create the comparison with replace(saved_config, noise_high=0.1).

SolutionVerify replay before changing one setting

Reconstruct and compare in a fixed order: configuration first, exact discrete identity next, and tolerance-aware numerical output last. When agreement fails, locate the first mismatch and trace it backward. When the goal is a controlled comparison, change exactly one named setting and report that change before interpreting the different result.

References

Python documentation: dataclasses.replace — creating a new dataclass instance with named field changes.
NumPy documentation: numpy.array_equal — exact array equality.
NumPy documentation: numpy.allclose — tolerance-aware numerical comparison.