Inspect a Text Corpus

Project question: Can you inspect a small directory of text documents, preserve each document's identity, measure what is actually present, and produce a report another program can reproduce?

What makes this a project

Text inspection begins with the bytes on disk. In this project you will verify the supplied file inventory, decode UTF-8 strictly, make line boundaries explicit, calculate deterministic measurements, apply configured inspection rules, select traceable examples, create numerical and visual summaries, and save a report that can be replayed.

The project teaches inspection before transformation. A split-on-whitespace count is not treated as a linguistic truth, and a frequent code point is not automatically important. Keep the source identity and the boundary of each measurement visible.

You will produce:

• a verified document inventory with accepted and rejected states;
• byte-first records and strict decoding results;
• explicit LF, CRLF, and lone CR line-boundary measurements;
• per-document and corpus measurements with conservation checks;
• configured findings and deterministic escaped examples;
• numerical and visual evidence with document identities;
• saved artifacts and a separate replay record; and
• a concise report whose claims stay within the inspection rules and fixture.

What you should know first

The project follows the Python path through Chapter 13. You should be able to work with paths and files, decode text under a stated encoding, validate tabular records, count and group values, keep IDs aligned with results, create plots, save structured artifacts, and replay a deterministic program. DATA-01 and DATA-03 are useful preparation but optional. The supplied corpus and configuration provide the local boundary if you did not complete either one.

Supplied corpus

The read-only inputs are:

data/
  documents.csv
  documents/
    ... text files ...
  corpus_manifest.json
plans/
  inspection_config.json

documents.csv contains unique document_id, safe relative path, and a neutral source_group. The manifest declares UTF-8, identifies every supplied file by byte count and SHA-256 digest, and records the fixture version. The small corpus includes empty and non-empty lines, non-ASCII characters, repeated content, and more than one line-ending style. A separate public fixture contains invalid UTF-8 bytes.

Identity and decoding

Resolve every path beneath the supplied document directory. Reject absolute paths, parent traversal, missing files, directories, duplicate paths, unlisted files, and digest mismatches. Read bytes first and record their identity; only then decode them strictly as UTF-8. Do not replace undecodable bytes or guess a different encoding. A decoding failure becomes an explicit rejected document record and remains in corpus totals.

Lines and measurements

After successful decoding, recognize LF, CRLF, and lone CR boundaries explicitly. Record their counts before making one analysis representation whose boundaries are normalized to \n.

Use these definitions:

• an empty file has zero decoded characters;
• a line is each segment separated by a recognized boundary, with a final unterminated segment counted when it contains characters;
• a blank line is a counted line containing only Unicode whitespace; and
• line length is the Unicode code-point count after removing the boundary.

For every accepted document preserve byte count, decoded code-point count, line count, blank-line count, maximum line length, whitespace and non-whitespace code-point counts, non-ASCII code-point count, distinct code-point count, and a digest calculated from the original bytes. Prove that whitespace plus non-whitespace counts equal decoded code-point count. The project may display common code points with escaped names, but it does not call them words or tokens.

Findings and examples

The inspection configuration supplies thresholds and exact rules for empty documents, decoding failure, mixed line-boundary styles, overlong lines, disallowed control code points, leading or trailing Unicode whitespace in a line, and duplicate original-byte digests.

A finding is a reason to inspect a record, not proof that the text is wrong. Save one finding per document/rule pair with document ID, rule name, measured evidence, and a short configured explanation. Every listed document receives exactly one final state: accepted_clean, accepted_flagged, or rejected. The three state counts must sum to the complete manifest count.

For each applicable rule, select examples by document ID and line number, keep the configured number, escape control characters, preserve source location, and use the configured maximum excerpt length with an explicit truncation marker. Never copy a shortened excerpt back into the corpus.

The project workspace

project/
  README.md
  data/                              # supplied, read only
  plans/inspection_config.json       # supplied starting config
  src/config.py                      # reader implementation
  src/load.py                        # reader implementation
  src/lines.py                       # reader implementation
  src/measure.py                     # reader implementation
  src/inspect.py                     # reader implementation
  src/report.py                      # reader implementation
  src/main.py                        # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py              # supplied, read only

Keep loading, line analysis, measurement, rule evaluation, example selection, and reporting independently inspectable. Maintain one continuous workspace through every milestone.

Required evidence

The completed project contains:

• document_inventory.csv with identity, acceptance state, and reason;
• document_measurements.csv;
• corpus_summary.json and source_group_summary.csv;
• inspection_findings.csv and deterministic selected_examples.json;
• duplicate_content_groups.json;
• document-length, line-length, and finding-count figures plus their plot data;
• inspection_manifest.json and a replay agreement or mismatch record; and
• a concise corpus inspection report at report.md.

Checks should include unsafe, duplicate, missing, extra, and changed paths/files; empty, ASCII, non-ASCII, invalid UTF-8, and mixed-boundary inputs; terminated and unterminated final lines; hand-computed counts; conservation; multiple findings for one document; duplicate bytes under different IDs; deterministic example order and escaping; complete status accounting; no source mutation; figure-data checks; artifact read-back; replay; and an interpretive rubric that rejects unsupported claims about language, quality, representativeness, or model suitability.

Limits

This project does not teach automatic encoding detection, Unicode normalization, language identification, semantic cleaning, tokenization, stemming, stop-word removal, vocabulary construction, linguistic quality scores, corpus balancing, sampling policy, privacy review, copyright decisions, web crawling, streaming corpora, search, embeddings, or language-model training. TEXT-02 consumes accepted document records while retaining source IDs and original identity.

Review

The final review asks whether every listed file is accounted for, bytes and decoding states are preserved, line rules are explicit, measurements conserve their totals, findings and examples are traceable, plots use saved identities, replay is separate, and the report distinguishes observations from claims about language or quality.