Milestone 8 of 9

Serialize, reload, and replay the workbench

Save and read back weights, queries, scores, rankings, judgments, measures, comparisons, figures, configuration, identities, and manifest, then replay separately.

The workbench is reproducible only when it saves its rules and evidence, not just its final scores. Read every artifact back, then rerun in a separate tree.

Goal

Serialize weights, norms, queries, contributions, rankings, judgments, traces, measures, comparisons, figures, configuration, identities, and a manifest; read every artifact back; and reproduce the complete analysis independently.

Inputs

Use all validated and generated artifacts from the earlier milestones, including the versioned retrieval configuration, corpus/tokenizer/index/query identity, formulas, rounding, ties, cutoffs, tolerances, plot data, and source records. Replay must use a separate directory and a fresh process or clean in-memory state.

Deliverables

Extend src/main.py and reporting helpers to save:

retrieval_workbench_manifest.json with artifact paths, schema and formula versions, source identities, counts, ordering rules, and configuration;
all required CSV, JSON, JSONL, figure, and exact plot_data.json artifacts;
read-back checks for every artifact; and
a replay agreement record, or a first-mismatch record naming artifact, path, expected value, and observed value.

The replay tree must contain a complete independently generated workbench, not a copy of the first output directory.

Checks

Read every artifact with its declared format and compare semantic records, identities, order, counts, scores, norms, judgment joins, denominators, differences, macro rows, and plot data. Check deterministic CSV, JSON, and JSONL output across repeated runs. Change one controlled cutoff or formula in a separate test and confirm the first meaningful mismatch is reported.

Reject corrupt, missing, incompatible, or reordered artifacts with an actionable reason. Confirm replay does not mutate inputs or the original output tree and does not claim agreement after a mismatch.

Workspace

Keep serialization, read-back, and replay comparison in src/main.py and clearly separated helpers. Write the first run to output/ and the independent run to a tree such as replay/. Keep every path and schema version in the manifest.

Hints

HintManifest assumptions first

Record identities, formulas, rounding, ties, cutoffs, tolerances, and orders before artifact paths. A file can parse correctly and still be incompatible.

HintCompare values after reading

Use declared schemas and semantic records, then compare ordering and values. File existence or byte equality alone does not prove a replay.

HintMake the first failure useful

Walk artifacts in manifest order and report the first path and expected/observed values. A mismatch is evidence, not a reason to silently rebuild the reference.

Review

Use the manifest to locate one score contribution, rank trace, cutoff record, and plot value. Then alter one configuration field and inspect the mismatch. Does the replay prove the same experiment ran, or only that files existed?

How to check your work

Checks compare the manifest, read-back checks, and replay result with the supplied fixtures. The supplied fixture regenerates artifacts in a separate tree and reports a first mismatch instead of copying or trusting the original outputs.