Milestone 1 of 9

Verify corpus, tokenizer, documents, and configuration

Check source identities, complete token positions, primary and audit settings, rational thresholds, evidence limits, document order, and short-document fixtures.

Before building groups, prove that the source documents, tokenizer, positions, and primary and audit rules belong together. A changed token policy defines a different analysis.

Goal

Validate corpus and tokenizer identity, complete document token records, manifest order, primary and audit settings, rational thresholds, evidence limits, and short-document inputs.

Inputs

Read accepted TEXT-02 token records, corpus identity, tokenizer identity, document manifest, plans/near_duplicate_config.json, and tiny fixtures. The configuration provides a positive non-boolean primary group size, an exact rational threshold, an ordered audit grid, evidence limits, plot order, and token-kind identity. The primary setting appears once in the audit grid, and audit settings are unique.

Deliverables

Implement validation in src/config.py. Produce an ordered validated input record containing corpus, tokenizer, source, and document identities; manifest order; tokenizer and selected-kind rule; primary and audit settings; rational thresholds; evidence limits; and plot order.

Checks

Reject changed corpus or tokenizer identity, duplicate or missing document IDs, changed corpus order, missing or duplicate token positions, altered token text, unsupported token kind, malformed records, and source-position gaps. Reject a non-positive or boolean-like group size, threshold numerator or denominator, numerator > denominator, duplicate audit settings, a missing primary setting, invalid evidence limits, and incompatible plot order.

Check empty, punctuation-only, shorter-than-group-size, exactly-sized, and longer documents, case-different, non-ASCII, underscore, and punctuation fixtures. Confirm input records are not mutated.

Workspace

Keep identity, configuration, ordering, and short-document checks in src/config.py. Do not select tokens, build groups, compare pairs, or create figures yet.

Hints

HintValidate the fraction as integers

Store numerator and denominator separately. The inequality 0 <= numerator <= denominator is the contract; a decimal approximation is not a validation rule.

HintPrimary is one audit row

Compare settings structurally, not by position alone. The primary setting must occur once, wherever it appears in the ordered grid.

HintShort is valid input

A document may have too few selected tokens for a group. Keep it as a document and classify it later; do not reject it as a bad source.

Review

Follow one document from manifest identity through its complete token rows. Then inspect a malformed threshold. Why would changing normalization or rounding before grouping invalidate a later pair decision?

How to check your work

Checks compare identities, positions, settings, thresholds, and limits with the fixtures. The supplied fixture rejects incompatible input instead of repairing it.