Scan canonical text without losing characters

Scanning should cover canonical text completely. Every code point belongs to one token and one span.

Goal

Scan canonical text left to right, emit typed tokens and half-open spans, prove complete non-overlapping coverage, and reconstruct each canonical document by joining token text.

Inputs

Use canonical document records. Emit:

• each \n as one newline token;
• each ASCII space as one space token;
• each maximal run of Unicode alphanumeric code points or underscore as one text token; and
• every remaining code point as one individual symbol token.

Each token stores text, kind, zero-based token position, and canonical-text span [start, end). Empty text produces no tokens.

Deliverables

Implement scanning in src/scan.py and save per-document token records with document ID, corpus identity, text, kind, position, start, and end. Include a reconstruction result for each document.

Do not hide whitespace tokens in the machine-readable record. Escape space, newline, controls, and other ambiguous text only in human-readable displays; the token record retains exact token text.

Checks

Use empty text, spaces, multiple newlines, leading/trailing whitespace, punctuation, underscores, non-ASCII letters, emoji, and combining marks. Check that the first span begins at zero, adjacent spans touch, the last span ends at canonical length, no span overlaps, and joining text reconstructs canonical text exactly.

Check token positions are contiguous, token kinds follow the supplied rule, and scanner output cannot produce one reserved string such as <unk> as a single token. Confirm input text and records are not mutated.

Workspace

Keep scanning and span construction in src/scan.py; use canonical text from src/normalize.py. Store token records under output/ and leave vocabulary IDs for the next milestone.

Hints

Review

Choose a canonical string with text, space, punctuation, and newline. Follow each span into the source and join the tokens back together. Explain why a token list without spans is weaker evidence.

How to check your work

Checks compare token kinds, spans, and reconstruction with the supplied fixture. The supplied fixture covers every canonical code point without calling a text token a linguistic word.