Milestone 7 of 8

Save and replay the toolkit

Write configuration, vocabulary, token records, encoded sequences, unknown traces, fixtures, source identity, and a manifest, then replay separately.

The toolkit should be usable from its saved configuration, vocabulary, records, and identity. Replay it separately before treating the artifacts as durable.

Goal

Write configuration, vocabulary, document token records, encoded sequences, unknown traces, summaries, fixtures, source identity, figures, and a manifest. Replay the toolkit in a separate directory and record agreement or mismatch.

Inputs

Use all artifacts from the earlier milestones and the original inspected corpus records. The manifest must identify the corpus, normalization and scanner versions, reserved IDs, vocabulary order and size, boundary policy, artifact paths, relevant runtime versions, and implementation revision.

Generate token-frequency-rank and per-document token/unknown-count figures from saved records and preserve their exact plot data. The replay starts from saved configuration and source identity under a separate directory such as output/replay/.

Deliverables

Extend src/main.py and src/report.py, or add a clearly separated replay function, to save:

  • output/tokenizer_config.json;
  • output/vocabulary.json and the escaped table;
  • per-document canonical and token records;
  • encoded sequences and output/unknown_tokens.csv;
  • document and corpus summaries;
  • exact normalization, scanning, vocabulary-order, codec, and round-trip fixtures;
  • figures and their plot data;
  • output/tokenizer_manifest.json; and
  • a replay agreement or mismatch record under output/replay/.

The manifest should reference artifacts rather than copy every sequence into one record. Never overwrite the original output during replay.

Checks

Read every required artifact back and check identity, schema, configured order, contiguous IDs, span coverage, unknown traces, summaries, fixtures, and plot data. Replay an unchanged toolkit separately and compare each transformation and artifact. Confirm the original output remains unchanged.

Deliberately change normalization version, reserved order, one vocabulary ID, one token span, or one unknown trace and check that replay identifies the first difference. An incompatible identity or runtime must not be reported as exact agreement.

Workspace

Keep original artifacts under output/ and replay artifacts under a distinct child directory. Keep output/tokenizer_manifest.json and project-level report.md at stable paths.

Hints

HintReplay transformations in order
Compare corpus/configuration identity, canonical text, token records, vocabulary, encoded IDs, decoding, and plot data in that order.
HintPlot data is an artifact
A frequency figure must point to exact saved token counts and escaped identities, not a new counting pass hidden in plotting code.

Review

Open original and replay artifacts side by side. Explain what agreement establishes for this corpus and configuration, and why it does not establish that the tokenizer is best or beneficial to a model.

How to check your work

Checks compare manifest and replay evidence with the version-matched reference fixture. The supplied fixture makes any changed tokenizer identity explicit.