Milestone 2 of 8

Decode without hiding failures

Read bytes first, apply strict UTF-8 decoding, and preserve rejected records instead of silently replacing or dropping invalid bytes.

Once a file is identified, decode its bytes under the stated rule. A decoding failure is a result that must remain visible.

Goal

Read verified bytes, apply strict UTF-8 decoding, preserve accepted and rejected document records, and hand-check ASCII, non-ASCII, empty, and invalid-byte fixtures.

Inputs

Use the verified inventory and the original file bytes. The corpus manifest declares UTF-8. Decode with strict error handling; do not replace undecodable bytes, guess another encoding, or change the original digest.

The public fixtures include ordinary ASCII text, non-ASCII text, an empty file, and invalid UTF-8 bytes. Keep source document ID and path on every decoded or rejected record.

Deliverables

Implement strict decoding in src/load.py and add a stable accepted/rejected document record for every listed document. Store normalized analysis text only for a successfully decoded document; retain byte count and digest for all records.

Record the decoding state and a short explicit error for an invalid file. Do not create a replacement text value for a failed decode, and do not remove the failed file from the manifest count.

Checks

Check ASCII, non-ASCII, empty, and invalid UTF-8 fixtures. An empty file is a successful decode with zero decoded characters; invalid bytes are a rejected document with no analysis text.

Check that accepted decoded text re-encodes to the original bytes under strict UTF-8, that document identity is unchanged, and that original bytes and inventory rows are not mutated. Read the failure record back and verify its error state.

Workspace

Keep byte loading and strict decode in src/load.py; keep line handling for the next milestone in src/lines.py. Store accepted/rejected records under output/ without writing decoded text back into data/.

Hints

HintEmpty is not failed

An empty byte string decodes successfully. Give it zero decoded characters rather than an error message.

HintStrict means no substitution

A replacement character would hide which bytes were invalid. Let strict decoding produce an explicit rejection.

Review

Compare an accepted non-ASCII record and an invalid-byte record. Explain why the first can proceed to line analysis while the second remains in corpus totals but cannot produce measurements based on decoded text.

How to check your work

Checks compare decode states, byte identity, and failure evidence with the supplied fixture. The supplied fixture does not guess an encoding for the reader.