Verify tokenizer, corpus, documents, and configuration

Begin with identity. A sequence builder should refuse incompatible source artifacts before it creates a single example.

Goal

Validate the tokenizer and corpus identities, document order, encoded IDs, boundary symbols, group assignments, context and batch sizes, empty documents, and the fixed remainder policy.

Inputs

Read the versioned tokenizer configuration and vocabulary, encoded documents in manifest order, plans/document_groups.csv, plans/sequence_config.json, and the supplied tiny fixtures. The vocabulary reserves IDs 0 through 3 for <pad>, <unk>, <bos>, and <eos> in that order. context_length and batch_size are positive integers, and the remainder policy is exactly keep.

A non-empty document must begin with one <bos>, end with one <eos>, contain neither internally, and contain no <pad>. <unk> is valid. An empty source document has an empty sequence and creates no examples, but its ID and group remain in summaries.

Deliverables

Implement validation in src/config.py and src/documents.py (or a clearly separated loader). Produce an ordered, identity-rich input record containing document ID, manifest position, corpus/tokenizer identity, encoded length, group, context length, batch size, group order, and remainder policy.

Checks

Reject a changed corpus or tokenizer identity, duplicate or missing document IDs, changed manifest order, unknown IDs, negative IDs, booleans, and IDs not present in the vocabulary. Reject missing, repeated, internal, or swapped boundary symbols and any source <pad>. Reject unknown document IDs, duplicate group assignments, missing assignments, unsupported group names, non-positive sizes, and any remainder policy other than keep.

Check an empty document, an empty group, and a document containing <unk>. Confirm that source records, configuration, group assignments, and vocabulary are not mutated.

Workspace

Keep identity and configuration checks in src/config.py and document checks in src/documents.py. Leave data/ and supplied plans read only. Do not create examples or batches in this milestone.

Hints

Review

Choose one non-empty and one empty document. Can a later milestone identify both without inventing a boundary pair or an example? Follow one rejected fixture back to the exact rule that rejected it.

How to check your work

Checks compare identity, boundary, group, and size failures with the version-matched fixtures. The supplied fixture refuses incompatible inputs instead of repairing them.