Milestone 7 of 8

Serialize, reload, and replay the engine

Save and read back the positional view, queries, traces, matches, excerpts, summaries, figures, configuration, identities, and manifest, then replay separately.

A result is more useful when another run can recover the same evidence. Save the assumptions as well as the answers, then replay from a separate tree.

Goal

Serialize the positional view, query analyses, traces, matches, excerpts, summaries, figures, configuration, identities, and manifest; read them back; and reproduce the complete result without reading the original output objects.

Inputs

Use all validated and generated artifacts from the earlier milestones, including source and tokenizer identity, TEXT-04 identity, rule versions, configuration, query order, plot data, and the complete output tree. The replay must use a separate directory and a fresh process or clean in-memory state.

Deliverables

Extend src/main.py and the relevant reporting modules to save:

phrase_proximity_manifest.json with artifact paths, schema and rule versions, source identities, counts, and deterministic ordering rules;
a versioned snapshot of phrase_proximity_config.json, corpus/tokenizer/ TEXT-04 index identity, and the query records used for the run;
JSONL, CSV, JSON, and figure artifacts from the earlier milestones;
artifact read-back checks for every saved artifact; and
a replay agreement record, or a first-mismatch record naming the artifact, path, expected value, and observed value.

The replay tree must contain a complete independently generated result, not a copy of the first output directory.

Checks

Read every artifact back with its declared format and compare records, identities, order, counts, ranges, trace evidence, summaries, and plot data. Check deterministic JSONL and manifest output across repeated runs. Change one controlled input in a separate test and confirm the replay reports the first meaningful mismatch rather than claiming agreement.

Check corrupt, missing, incompatible, or reordered artifacts are rejected with an actionable reason. Confirm the original inputs and first output tree are not mutated during replay.

Workspace

Keep serialization and read-back logic in src/main.py and clearly separated helpers. Write the first run to output/ and the independent run to a replay-specific tree such as replay/. Keep all paths and schema versions in the manifest.

Hints

HintManifest the assumptions

Record the identities and rule versions before the artifact paths. A file can parse correctly and still be incompatible with the engine.

HintCompare structure before bytes

Read values into the declared schemas and compare semantic records and ordering. Byte equality may be a useful extra check, but it is not a substitute for read-back validation.

HintMake failure useful

Walk artifacts in manifest order and stop at the first difference, reporting its path and values so a reader can fix the replay rather than guess.

Review

Open the manifest and use it to locate one trace, one excerpt, and the plot data. Then alter a single query limit and inspect the mismatch record. Does the replay prove the same engine and rules ran, or only that files existed?

How to check your work

Checks compare the manifest, read-back checks, and replay result with the supplied fixtures. The supplied fixture regenerates artifacts in a separate tree and reports the first mismatch; it does not copy or trust the original objects.