Rerunning and Comparing Experiments

Repeat a randomized computation from its recorded seed and configuration, then compare the new result with the saved evidence. State which parts should match exactly, which require numerical tolerance, and which environmental differences remain outside the seed's control.

A saved result is more useful when the computation can be run again. A rerun executes the experiment again from a recorded configuration, code revision, and input-data identity.

The purpose of a rerun is not always to produce textually identical output. It is to test a stated expectation:

• Should the generated sample match exactly?
• Should a numerical measurement be equal within a tolerance?
• Should only one controlled configuration change explain the difference?
• Should a relationship or conclusion remain the same?

State the expected kind of agreement before comparing results.

Reconstruct the original run

Suppose a saved record contains:

Rebuild the configuration from the saved fields rather than relying on current defaults:

Then call the same experiment function:

rerun_measurements = run_coin_experiment(config)

If the code, generator behavior, call sequence, and input values match, this small computation should reproduce its generated values and measurements.

Hard-coding only the saved seed is insufficient if another setting has changed. A rerun begins from the complete recorded configuration.

Compare exact discrete results exactly

For integer arrays, Boolean arrays, indexes, labels, and other discrete values, use exact equality when exact replay is expected:

For ordinary Python dictionaries containing exactly comparable values:

same_record = original_record == rerun_record

Do not compare NumPy arrays with a dictionary equality operation that expects one Boolean result. Use an array comparison method suited to the value type.

An exact mismatch is evidence that some required condition differs. It does not identify the cause by itself.

Compare floating-point results with a stated tolerance

Floating-point calculations can differ slightly across implementations, operation orders, package versions, or hardware. When small numerical differences are acceptable, use np.isclose for scalars and np.allclose for arrays:

The comparison accepts values when:

absolute difference
<=
absolute tolerance + relative tolerance * comparison scale

• atol is the absolute tolerance. It matters near zero.
• rtol is the relative tolerance. It scales with the magnitude being compared.

Choose tolerances from the numerical meaning and expected computation. Do not increase them until a failing result passes. A tolerance that accepts a scientifically or practically meaningful change is too loose.

When the same deterministic operations are expected to replay bit for bit, exact equality may still be appropriate for floating-point arrays. State that stronger expectation explicitly.

Compare configuration before measurements

When results differ, first determine whether the runs were intended to be the same:

If the configuration differs, identify each changed field:

A comparison with several changed settings cannot isolate which change caused the result. For a controlled experiment, create a baseline and change one intended factor while holding the others fixed.

Configuration equality is necessary but may not be sufficient. Data, code, and environment can still differ.

Verify a complete rerun

The following demonstration records one simulation and runs it again. Change one field in rerun_config and inspect which checks fail.

The checks proceed from input description to generated values to derived measurements. This order helps locate the first disagreement.

Diagnose a mismatch in dependency order

When a rerun differs, check the dependencies that produced the result:

1. Configuration: Do seed, sample size, split rule, and other settings match?
2. Input data: Is the same data version present, with the same order and preprocessing?
3. Code: Is the same implementation or source revision running?
4. Random call sequence: Did the generator receive the same calls in the same order?
5. Environment: Are relevant Python, NumPy, library, hardware, and runtime details compatible?
6. Comparison rule: Is exact or approximate equality appropriate, and is the tolerance justified?

Begin with the earliest difference. A later measurement mismatch may be a consequence of an earlier configuration or data mismatch.

Avoid the vague conclusion “randomness changed it.” Name the changed input, call, environment, or comparison assumption when the evidence allows it.

Distinguish expected variation from a failed replay

Two experiments may deliberately use different seeds to estimate how a result varies across random samples:

These runs are not failed reruns of one sequence. They are separate configured runs designed to explore seed-dependent variation.

Record every seed and summarize results across runs. Do not repeatedly run without recording seeds and then keep only the most favorable result.

Use two different questions:

• Replay question: Can this recorded run be reconstructed?
• Variation question: How much does the result change across intentionally different random runs?

Both matter, but they require different comparisons.

Compare conclusions as well as files

Exact files can differ while a higher-level result remains stable. For example, five configured seeds may produce different simulated samples but similar mean errors.

Preserve both levels of evidence:

run-level evidence:
seed, configuration, exact sample or artifact, measurements

across-run evidence:
measurement distribution, range, mean, uncertainty, stable or unstable claim

The later project chapter develops a small experiment report. At this stage, the important habit is to avoid replacing run-level evidence with one unexplained aggregate.

configuration matches: True
sample matches: True
mean close: True

Retain the rerun comparison order

Use this order when verifying or diagnosing a rerun:

reconstruct configuration
-> identify data, code, and environment
-> repeat random calls
-> compare generated values
-> compare derived measurements
-> compare the stated conclusion

At the first mismatch, inspect the dependency that produced it. Reproducibility is not one Boolean property of an experiment; it is a set of explicit expectations checked against preserved evidence.