Milestone 2 of 8

Normalize under one explicit rule

Preserve newlines and all non-whitespace code points while collapsing each run of other Unicode whitespace to one visible ASCII space.

Normalization creates a new representation. Record exactly what changed and never present canonical text as the original source.

Goal

Produce canonical text under the supplied whitespace rule, retain document and corpus identity, and record whether and how much each document changed.

Inputs

Use TEXT-01 analysis text whose line boundaries are already \n. Scan from left to right and:

  1. preserve every \n exactly;
  2. replace each non-empty run of other Unicode whitespace with one ASCII space;
  3. preserve every other code point and its case exactly.

Do not trim, lowercase, remove punctuation, replace accents, or apply Unicode normalization. Empty input remains empty.

Deliverables

Implement normalization in src/normalize.py. Return canonical text with document ID, corpus identity, a changed/not-changed field, the number of collapsed whitespace code points or runs under one documented definition, and enough evidence to compare analysis and canonical lengths.

Save per-document normalization records for later artifacts. Preserve the analysis text separately; never write canonical text back over the source.

Checks

Hand-check spaces, tabs, multiple non-newline whitespace characters, multiple newlines, non-ASCII text, punctuation, leading and trailing whitespace, and empty input. Verify every newline survives exactly and every other non-empty whitespace run becomes one ASCII space.

Check identity preservation, deterministic results, and no mutation of analysis text. Confirm that no case, accent, punctuation, emoji, or combining mark is changed by the supplied rule.

Workspace

Keep normalization in src/normalize.py; consume validated documents and configuration from the first milestone. Store canonical records under output/ only when the runner assembles the project artifacts.

Hints

HintNewlines are their own case
Handle \n before the rule for other Unicode whitespace so it cannot collapse into an ASCII space.
HintRecord both representations
A changed flag is useful only when the source analysis text and canonical result remain separately identifiable.

Review

Choose a document with tabs or leading whitespace and compare its analysis and canonical text. Explain what changed, what stayed exact, and why canonical text is not the unmodified source.

How to check your work

Checks compare edge-case normalization and change counts with the supplied fixture. The supplied fixture implements only the supplied rule.