Milestone 2 of 9

Calculate document frequencies, weights, and norms

Build sparse non-zero document weights, retain zero-term documents, and prove term-count, posting, smoothed natural-log idf, and norm invariants.

The cosine system needs sparse vectors with a documented scale. Build them from term counts and retain the arithmetic that makes every norm inspectable.

Goal

Calculate document frequencies, smoothed natural-log idf values, sparse non-zero document weights, and document norms while preserving zero-term documents and TEXT-04 count invariants.

Inputs

Use the validated corpus and positional inverted index. Let N be the positive number of corpus documents, including documents with no searchable terms. For each indexed term t:

document_frequency(t) = number of corpus documents containing t
idf(t) = ln((N + 1) / (document_frequency(t) + 1)) + 1
document_weight(t, d) = document_term_count(t, d) * idf(t)
document_norm(d) = sqrt(sum(document_weight(t, d) ** 2 for all indexed t))

Use the natural logarithm. Store document frequency, idf, every non-zero document weight, and every document norm. Keep every document and a zero norm for a document with no searchable terms. Do not build a dense document-by-vocabulary matrix merely for convenience.

Deliverables

Implement src/weights.py. Produce output/term_statistics.csv with term, document frequency, and idf; output/document_weights.csv with every non-zero weight; and output/document_norms.csv with every document and norm. Include formula and identity versions in the artifact metadata or manifest-ready records.

Checks

Hand-calculate frequencies, smoothed idf, weights, and norms on a tiny corpus. Check each term's document frequency equals its posting count, each document term count agrees with TEXT-04, and each norm is the square root of its saved weight squares. Check finite non-negative values, natural-log behavior, exact term and document order, repeated terms, shared terms, non-ASCII terms, underscores, and an empty document with norm zero.

Check that no dense vocabulary matrix is required, all source/index inputs stay unchanged, and a term occurring in every document still receives the supplied smoothed idf rather than an invented zero.

Workspace

Keep frequency, idf, sparse weight, and norm construction in src/weights.py. Write only the three statistics artifacts and their hand-check evidence. Do not analyze query vectors or score systems yet.

Hints

HintFrequency counts documents

A posting count answers how many documents contain a term; an occurrence count answers how often it appears. The idf formula uses the first quantity.

HintKeep the zero case

Start document norms from every document in manifest order. Missing term entries should leave a document's norm at zero, not remove the document.

HintSquare the saved weights

For a hand check, list each non-zero weight, square it, sum the squares, and take one square root. This is the document norm contract exactly.

Review

Choose one term and one document and trace its posting count, idf, weight, and contribution to the norm. What changes when a term appears in every document, and what must remain true for an empty document?

How to check your work

Checks compare the three sparse artifacts with the hand-check fixture. The supplied fixture uses the stated smoothing and natural logarithm, retains zero-term documents, and does not replace sparse records with a dense vocabulary table.