Load the corpus and configuration

Begin with the corpus and tokenizer identities. The transformation rules must be fixed before any text is changed.

Goal

Verify corpus and tokenizer identity, reserved symbols, vocabulary limits, document order, and empty-document behavior before normalizing text.

Inputs

Use accepted document records and the versioned corpus manifest from TEXT-01 or the supplied fallback, plus plans/tokenizer_config.json. The configuration contains normalization version, scanner version, required reserved-symbol order, maximum vocabulary size, and boundary-token policy.

The reserved order is exactly <pad>, <unk>, <bos>, <eos>. The maximum vocabulary size must be at least four. Each document retains a unique document_id, manifest position, source identity, and analysis text. An empty document remains an empty document record.

Deliverables

Implement loading and validation in src/config.py and src/main.py or a small dedicated loader. Produce a normalized configuration record and an ordered document inventory without transforming the analysis text.

Record corpus-manifest identity, tokenizer configuration identity, document order, reserved IDs, maximum size, and boundary policy in the report or a stable project record.

Checks

Reject mismatched corpus identity, repeated document IDs, changed document order, duplicate reserved strings, changed required reserved order, too-small vocabulary size, and unsupported version values. Check an empty corpus record and an empty document without inventing tokens or boundary IDs.

Check that the source document records and configuration are not mutated and that every later output can retain both document and corpus identity.

Workspace

Keep configuration validation in src/config.py and orchestration in src/main.py. Leave data/ and supplied plans read-only. Do not write canonical text, tokens, or vocabulary artifacts in this milestone.

Hints

Review

Read the accepted configuration as a later consumer. Can you identify the corpus, document order, all reserved IDs, maximum vocabulary size, and boundary policy without opening the implementation?

How to check your work

Checks compare configuration validation and identity evidence with the supplied fixture. The supplied fixture does not silently migrate an incompatible tokenizer identity.