Milestone 3 of 8

Make line boundaries explicit

Count LF, CRLF, and lone CR boundaries before creating one normalized analysis representation and checking edge cases.

Line counts depend on the boundary rule. Count the original forms before normalizing them for later analysis.

Goal

Recognize LF, CRLF, and lone CR boundaries explicitly, record their counts, create normalized analysis text with \n boundaries, and verify the supplied edge cases.

Inputs

Use successfully decoded document records and their original text. A boundary is one of LF, CRLF, or lone CR; count CRLF as one boundary, not two. Public fixtures include empty files, one unterminated line, a trailing boundary, mixed boundary styles, and non-ASCII text.

Use these line definitions:

  • an empty file has zero decoded characters;
  • a line is each segment separated by a recognized boundary, with a final unterminated segment counted when it contains characters;
  • a blank line is a counted line containing only Unicode whitespace; and
  • line length is Unicode code-point count after removing the boundary.

Deliverables

Implement boundary analysis in src/lines.py. For each successfully decoded document, save LF, CRLF, and lone-CR counts, normalized analysis text, line records or enough information to reconstruct them, and source identity.

Keep the original decoded text available for comparison. Normalization is an analysis representation only; do not write it over the source or change the original-byte digest.

Checks

Check an empty file, one unterminated line, a trailing boundary, mixed styles, and non-ASCII line content. Verify that CRLF counts once, lone CR is not lost, normalized boundaries are \n, and a final segment is counted only when it contains characters.

Check blank-line and line-length definitions with Unicode whitespace and that source identity and decoded code points are preserved. Ensure line analysis does not mutate the accepted document record or original text.

Workspace

Keep boundary parsing and line records in src/lines.py; consume strict decoded records from src/load.py. Store analysis evidence under output/ and leave measurement and inspection rules for later modules.

Hints

HintCheck CR before LF
When scanning a CR, look at the next code point. A following LF is one CRLF boundary; otherwise it is a lone CR.
HintKeep two representations
The original decoded text preserves what was read. The normalized text makes later line calculations consistent. They have different roles.

Review

Trace one mixed-boundary fixture from its original text to boundary counts and normalized text. Explain why a line count without the boundary counts is hard to audit.

How to check your work

Checks compare edge-case line counts and normalized output with the supplied fixture. The supplied fixture makes final unterminated segments explicit.