Measure each document and the corpus

Measurements describe the accepted records. Keep the document identity beside each count so a corpus total can be traced back to its source.

Goal

Calculate the defined per-document measurements, prove conservation invariants, aggregate corpus totals and source-group summaries, and retain stable document identity throughout.

Inputs

Use accepted decoded document records and normalized line analysis. For each accepted document preserve byte count, decoded code-point count, line count, blank-line count, maximum line length, whitespace and non-whitespace code-point counts, non-ASCII code-point count, distinct code-point count, and the digest of the original bytes.

Use source_group only to form descriptive summaries. Do not call code points words or tokens, and do not infer language or quality from a count.

Deliverables

Implement measurements in src/measure.py and save:

• output/document_measurements.csv with one row per accepted document;
• output/corpus_summary.json with corpus totals and state-aware counts; and
• output/source_group_summary.csv with group counts and defined totals.

Keep rejected decode records in the inventory and corpus totals even though they have no decoded-text measurements. Make the original-byte digest part of each accepted measurement record.

Checks

Check each measurement against a hand-counted ASCII, non-ASCII, blank-line, and mixed-boundary fixture. Prove that whitespace plus non-whitespace counts equal decoded code-point count for every accepted document and for the corpus sum.

Check that measurement row count and identities match accepted decoded records, source-group totals conserve accepted records, and all input records and normalized text remain unchanged.

Workspace

Keep count logic in src/measure.py; read identity and normalized lines from src/load.py and src/lines.py. Write CSV and JSON artifacts under output/; do not apply inspection rules in this module.

Hints

Review

Choose one document measurement row and trace its byte count, digest, line counts, and code-point counts to the source. Explain why corpus totals need accepted identities rather than only one large sum.

How to check your work

Checks compare measurement columns, conservation checks, and group summary fields with the supplied fixture. The supplied fixture does not assign linguistic importance to a frequency.