Project 10

Build a Document Search Index

Map exact text terms to documents and positions, then answer supplied queries with visible integer scores, deterministic rankings, and traceable excerpts.

  • 8 milestones
  • Optional
  • Browser workspace

Project question: Can you build a small search index that shows which exact terms occur in which documents, then explain every returned result?

What makes this a project

Search begins with a record of the source, not with a polished result page. In this project you will turn token records into an inverted index, keep every matching position, process queries with the same rules as the index, and rank results with a supplied arithmetic rule. The saved evidence will let you trace one result back to the terms that produced its score.

This is a bounded lexical search system. It can find exact text under an explicit rule. It does not understand synonyms or decide whether a result is useful to a person. Later retrieval projects can replace parts of this system after this boundary is clear.

You will produce:

  • a checked source, tokenizer, and search configuration;
  • an inverted index with full token positions and conservation checks;
  • query records that retain repeated, ignored, and missing terms;
  • scores with one contribution row for every query term, including zeroes;
  • deterministic results and source-token excerpts;
  • a serialized index, reload check, and separate replay; and
  • a concise search-index card that stays within the evidence.

What you should know first

The project follows the Python path through Chapter 13. You should be able to validate records, use dictionaries and lists, count repeated values, preserve positions, sort with exact ties, work across several files, save structured artifacts, and test deterministic programs. TEXT-01 and TEXT-02 are useful but optional. If you skipped them, use the supplied version-matched fallback records; do not invent a new tokenizer.

Supplied inputs

Use accepted documents and complete token records in corpus-manifest order, the exact tokenizer configuration that produced them, plans/search_config.json, plans/queries.jsonl, and the tiny hand-check fixtures. The configuration selects text tokens, sets a positive top_k and excerpt radius, and names the ranking-rule version. Every artifact keeps corpus and tokenizer identity.

The project workspace

project/
  README.md
  data/                              # supplied, read only
  plans/search_config.json           # supplied starting config
  plans/queries.jsonl                # supplied, read only
  src/config.py                      # reader implementation
  src/documents.py                   # reader implementation
  src/index.py                       # reader implementation
  src/query.py                       # reader implementation
  src/rank.py                        # reader implementation
  src/excerpts.py                    # reader implementation
  src/report.py                      # reader implementation
  src/main.py                        # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py              # supplied, read only

Keep source validation, index construction, query processing, scoring and ranking, excerpt selection, and serialization independently inspectable. Keep one workspace through every milestone.

Project milestones

Work through these in order. Each milestone produces evidence used by the next one, while project completion remains separate from lesson progress.

  1. 1Verify source, tokenizer, and search configuration

    Check corpus and tokenizer identity, complete token positions, manifest order, exact scanner rules, positive result limits, excerpt bounds, and supplied query identities.

  2. 2Build and hand-check a tiny inverted index

    Map each exact text term to manifest-ordered document postings and strictly increasing full-token positions, then compare every count with a hand calculation.

  3. 3Index the complete corpus deterministically

    Build lexically ordered terms, preserve zero-term documents, and prove indexed-length and posting-count conservation across the accepted corpus.

  4. 4Process queries with the same token rules

    Normalize and scan each supplied query exactly like the corpus while retaining repeated text terms, ignored non-text tokens, missing terms, and explicit empty-result reasons.

  5. 5Score and rank with complete contribution traces

    Add exact query-count and document-count products, record zero and positive contributions, apply fixed tie rules, and return no zero-score filler results.

  6. 6Build deterministic result excerpts

    Locate the earliest matching full-token position, take the configured bounded token window, and record ranges, matches, and truncation without changing source text.

  7. 7Serialize, reload, and replay search

    Save the index, statistics, analyses, results, excerpts, plots, and manifest; read them back separately; and reproduce the same ranked evidence.

  8. 8Write and audit the search-index card

    State exact indexing, query, scoring, ranking, excerpt, identity, and replay rules without claiming semantic relevance or general search quality.

Required evidence

Save inverted_index.json, document_statistics.csv, term_statistics.csv, query_analysis.jsonl, ranked_results.jsonl, score_contributions.csv, result_excerpts.jsonl, the indexed-length, posting-count, query-candidate, and returned-result figures with exact plot data, search_index_manifest.json, reload checks, a replay agreement or mismatch record, and report.md as the search-index card.

Limits

This project does not teach natural-language relevance, Boolean syntax, phrases, proximity, spelling correction, synonyms, lowercasing, stemming, stop-word removal, TF–IDF, BM25, probability, embeddings, vector databases, approximate search, query expansion, reranking, click feedback, web crawling, access control, production indexing, or performance benchmarking. It does not claim that exact term frequency measures meaning, fairness, popularity, or universal relevance.

Review

The final review asks whether identity and document order remain intact, postings contain every full token position, counts agree at every level, queries use compatible rules, ignored and missing terms remain visible, every score is reproducible from its contribution rows, ties are deterministic, zero-score documents are excluded, excerpts preserve source tokens, replay is separate, and the card distinguishes observations from claims.