Milestone 3 of 8

Index the complete corpus deterministically

Build lexically ordered terms, preserve zero-term documents, and prove indexed-length and posting-count conservation across the accepted corpus.

Once the tiny example is clear, apply the same construction to every supplied document. Determinism means that a second run has the same records, not merely the same number of results.

Goal

Build the complete positional inverted index while retaining document order, term spelling, case, non-ASCII text, underscores, zero-term documents, and stable summary records.

Inputs

Use the validated token records in corpus-manifest order. Select only records whose kind is text. Do not lowercase, stem, remove common terms, join neighboring tokens, or replace text with vocabulary IDs. A term that TEXT-02 encoded as <unk> still remains indexable here because this index uses the original token text.

Deliverables

Extend src/index.py and src/report.py to produce:

output/inverted_index.json with deterministic term and posting order;
output/document_statistics.csv, including zero-term documents; and
output/term_statistics.csv with corpus and posting counts.

Record corpus and tokenizer identity alongside the index. Keep full token positions, not character offsets and not only the first matching position.

Checks

Re-run every tiny-corpus invariant over the complete corpus. Check that terms are lexicographically ordered, postings follow manifest order, positions are strictly increasing, and every count agrees. Reconstruct each source document from its token records and check that indexable length counts exactly the selected text records.

Use equal-frequency terms to check order, repeated terms to check positions, case-different terms to check identity, and a document with no indexable terms to check accounting. Run the builder twice and compare complete JSON records, not just aggregate totals. Confirm inputs remain unchanged.

Workspace

Keep complete-corpus construction and summaries in src/index.py and src/report.py. Query analysis belongs to the next milestone. Do not add ranking or excerpts here.

Hints

HintOrder is part of the artifact

A dictionary may preserve insertion order, but insertion order is not the accepted term order. Apply the stated sort keys before writing.

HintKeep an empty document visible

The index has no posting for an empty document, but document statistics must still name and count it.

Review

Pick one term with several occurrences in several documents. Can you locate all of its positions without inspecting the source text again? What does the complete-corpus conservation check tell you, and what does it not tell you about search quality?

How to check your work

Checks compare the complete index and summaries with the supplied fixture. The reference preserves exact source identity and uses no undocumented linguistic rule.