Measure, save, reload, and replay the evidence

Byte counts answer different questions. Measure the controlled integer streams separately from complete file sizes, then save and replay every artifact.

Goal

Calculate fair numerical size accounting, save all compression and equivalence evidence, read it back, and reproduce the binary file, rebuilt index, searches, and measurements in a separate replay tree.

Inputs

Use the canonical index.bin, source JSON bytes, gap and varint traces, source and rebuilt indexes, search comparisons, configuration, identities, and exact plot-data rules.

Report complete index.bin bytes and source JSON bytes as labelled representation sizes, never as a controlled compression ratio. For the fair numeric comparison, use the same sequence of posting values. The fixed-width baseline stores each posting count, absolute document ordinal, position count, and absolute full-token position as an unsigned four-byte big-endian integer. The compressed stream stores the same posting counts plus document and position gaps using the canonical variable-byte rule.

Exclude document/term strings, outer headers, and unrelated JSON punctuation from both numerical streams. The supplied values fit the unsigned four-byte range.

Deliverables

Implement measurement and orchestration in src/measure.py and src/main.py. Produce output/size_accounting.csv with value count, baseline bytes, compressed bytes, signed byte difference, and compressed_bytes / baseline_bytes. Save figures from exact output/plot_data.json.

Save compressed_index_manifest.json with artifact paths, identities, schema and rule versions, bounds, counts, deterministic ordering, digests, and replay metadata. Include every required artifact, read-back checks, and a separate replay agreement or first-mismatch record, including the malformed-stream results from the parser and container milestones. The replay tree must be independently generated, not copied from output/.

Checks

Check complete file sizes and controlled stream sizes use their labelled denominators. Check value counts, four-byte arithmetic, canonical gap bytes, signed differences, ratios (including a ratio above one), exact plot data, and deterministic figures. Do not claim measured space or speed gains beyond the fixture.

Read every CSV, JSON, JSONL, binary, digest, comparison, and figure-data artifact back. Reproduce index.bin, rebuilt structure, searches, and size rows in a fresh replay tree. Change one controlled input and report the first mismatch; reject corrupt, missing, reordered, or incompatible artifacts.

Workspace

Keep size accounting in src/measure.py and orchestration, manifest, read-back, and replay in src/main.py. Write the first run under output/ and the second under replay/ or another separate tree. Do not mutate source artifacts.

Hints

Review

Explain why a ratio above one is valid. Then open the manifest and locate one size row, one structural comparison, one search comparison, and its replay evidence. Which denominator supports each claim?

How to check your work

Checks compare size rows, plot data, manifest, read-back checks, and replay outcome with the fixtures. The supplied fixture labels representations honestly and reports the first mismatch instead of repairing it.