Milestone 9 of 10

Measure, audit limits, save, reload, and replay

Separate body, payload, and container sizes; aggregate corpus totals; compare bounded dictionary limits; save exact artifacts and figures; and reproduce them separately.

Size claims need named denominators, and reproducibility needs saved rules as well as outputs. Measure both, then replay from a separate tree.

Goal

Calculate honest per-document and corpus size records, audit dictionary limits, save figures from exact plot data, read every artifact back, and reproduce the analysis independently.

Inputs

Use source inventory, codes, payloads, containers, round-trip results, primary and audit limits, configuration, and plot order. For each document and limit:

source_bytes
code_count
code_body_bytes = 2 * code_count
payload_bytes = 20 + code_body_bytes
container_bytes = 8 + payload_bytes
body_difference = source_bytes - code_body_bytes
stored_difference = source_bytes - container_bytes
body_ratio = code_body_bytes / source_bytes
stored_ratio = container_bytes / source_bytes

Ratios are null with an explicit reason for an empty source. Corpus totals sum numerator and denominator bytes before division; never average per-document ratios. Keep code body, payload, and complete container separate.

For every document and limit, report final dictionary size, additions attempted and accepted, code count, code-body bytes, and complete container bytes. Relative to primary, save signed changes. The audit does not choose a universally best limit or introduce reset heuristics.

Deliverables

Implement src/measure.py, src/report.py, and orchestration in src/main.py. Produce output/size_accounting.csv, corpus totals, output/dictionary_limit_audit.csv, exact output/plot_data.json, and figures. Save dictionary_codec_manifest.json with identities, rules, limits, artifact paths, counts, and deterministic order; read-back checks; and a separate replay agreement or first-mismatch record.

Checks

Check examples that become smaller, equal where possible, and larger; empty source null ratios; complete file/header overhead; correct corpus aggregation; code, payload, and container byte formulas; every configured limit; additions and final sizes; signed primary-relative changes; deterministic tables and plot data; and input immutability.

Read every CSV, JSON, JSONL, binary, digest, and plot-data artifact back. Run a fresh replay in a separate tree and reproduce primary and audit results. Change one source or limit in a separate replay and report the first mismatch. Do not call a character count, Python object size, or formatted artifact size a compression denominator.

Workspace

Keep size formulas and limit audit in src/measure.py, figures in src/report.py, and serialization/read-back/replay in src/main.py. Write the first run under output/ and the second under replay/.

Hints

HintKeep header cost visible

A 20-byte payload header and 8-byte outer header belong in complete container bytes, not in the code-body stream. Report both.

HintAggregate bytes, not ratios

Sum source and representation bytes across documents, then divide. An average of document ratios gives a different quantity.

HintNull is evidence

An empty source has no ratio denominator. Store null and an explicit reason instead of zero or an invented ratio.

Review

Explain a ratio above one and an empty-source null. Then use the manifest to locate one audit row, one plot value, one round-trip row, and replay evidence.

How to check your work

Checks compare size rows, corpus totals, audit records, plot data, manifest, read-back, and replay with the fixtures. The supplied fixture labels denominator choices and does not claim universal shrinkage.