Milestone 1 of 8

Inventory and verify the supplied corpus

Validate document identities and safe paths, then account for every missing, extra, duplicate, or changed file before decoding text.

Begin with the manifest and paths. Do not decode a document until you know which bytes the corpus says should be present.

Goal

Validate document IDs and safe paths, compare every supplied file with the manifest, and account for missing, extra, duplicate, or changed files before decoding text.

Inputs

Use the read-only data/documents.csv, data/documents/ directory, and data/corpus_manifest.json. Each inventory row contains a unique document_id, a safe relative path, and a neutral source_group. The manifest supplies expected byte counts, SHA-256 digests, and fixture version.

Reject absolute paths, parent traversal, empty or duplicate paths, missing files, directories where a file is expected, unlisted files, and changed byte count or digest. Keep document ID and path separate; neither is allowed to silently replace the other.

Deliverables

Implement the inventory boundary in src/load.py or a clearly separated function. Produce the first version of output/document_inventory.csv with document ID, path, source group, manifest identity, byte count, digest, and a clear accepted or rejected reason.

Record the complete manifest count and the counts of accepted, missing, extra, duplicate, and changed paths in the report or a stable inventory section. Do not decode or measure a file whose identity has not passed this boundary.

Checks

Use public fixtures for an unsafe path, duplicate path, missing file, extra file, changed bytes, and a directory in place of a file. Check that every manifest entry and every discovered file is accounted for and that each document ID is unique.

Check byte counts and SHA-256 digests from the original bytes, source order, and no mutation of the supplied files. A path that happens to resolve on the current machine is not safe if it escapes the supplied document directory.

Workspace

Keep path resolution, byte reading, and inventory checks in src/load.py. Leave data/ read-only and write inventory evidence under output/. Do not put decoded text or shortened excerpts into the inventory.

Hints

HintResolve beneath one root

Join a candidate path to the supplied documents directory, then verify that the resolved path remains beneath that root.

HintDigest the bytes you read

The manifest identifies original bytes. Calculate the digest from those bytes before decoding or normalizing anything.

Review

Choose one accepted and one rejected inventory row and follow each back to the manifest and filesystem. Explain why a rejected path remains evidence instead of disappearing from corpus totals.

How to check your work

Checks compare the inventory schema and path failures with the supplied fixture. The supplied fixture preserves identity before any text interpretation.