Project 11

Build a Phrase and Proximity Search Engine

Extend an exact-term positional index so that it can find ordered phrases and bounded term proximity while retaining the source positions that prove every match.

  • 8 milestones
  • Optional
  • Browser workspace

Project question: Can you extend an exact-term index so that every phrase and proximity result keeps the positions that prove it?

What makes this a project

Search becomes a different question when the order and distance between terms matter. In this project you will keep the source and tokenizer identity from an exact-term index, derive a searchable positional view, analyze structured queries, and find phrases and bounded term pairs. Every match will retain both its searchable-term positions and the full-token positions needed to inspect the source.

This is a bounded lexical search engine. It compares exact token text under an explicit rule. It does not understand meaning, repair spelling, or rank results by relevance. The saved traces and excerpts make those limits visible.

You will produce:

  • a checked corpus, tokenizer, positional-index, and query configuration;
  • a two-coordinate positional term index that conserves the TEXT-04 evidence;
  • query-analysis records that retain selected and ignored tokens;
  • complete phrase and proximity matches with candidate traces;
  • deterministic result order, source-linked excerpts, and exact plot data;
  • serialized artifacts, read-back checks, and a separate replay; and
  • a concise engine card that explains the evidence and the limits.

What you should know first

The project follows the Python path through Chapter 13. You should be able to validate structured records, use nested mappings and ordered sequences, scan two lists, preserve positions, sort with exact ties, work across files, save structured artifacts, and test deterministic programs. TEXT-04 is useful but optional. If you skipped it, use the supplied version-matched canonical fallback; do not invent a new tokenizer or index rule.

Supplied inputs

Use accepted documents, complete scanner-token records, corpus and tokenizer identities, and the TEXT-04 positional inverted index or its canonical equivalent. Also use plans/phrase_proximity_config.json, plans/queries.jsonl, and the tiny hand-check fixtures. The supplied rule indexes only text records and keeps exact token spelling, case, non-ASCII characters, and underscores. It does not lowercase, stem, remove common terms, correct spelling, or replace terms with IDs.

The project workspace

project/
  README.md
  data/                              # supplied, read only
  plans/phrase_proximity_config.json # supplied starting config
  plans/queries.jsonl                # supplied, read only
  src/config.py                      # reader implementation
  src/positions.py                   # reader implementation
  src/queries.py                     # reader implementation
  src/phrases.py                     # reader implementation
  src/proximity.py                   # reader implementation
  src/excerpts.py                    # reader implementation
  src/report.py                      # reader implementation
  src/main.py                        # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py              # supplied, read only

Keep one continuous workspace through every milestone. Position derivation, query analysis, phrase matching, proximity matching, excerpt construction, and artifact reporting should remain separately inspectable.

Project milestones

Work through these in order. Each milestone produces evidence used by the next one, while project completion remains separate from lesson progress.

  1. 1Verify corpus, tokenizer, index, configuration, and queries

    Check source identities, complete token positions, TEXT-04 posting compatibility, exact query records, rule versions, directions, limits, and zero-term documents.

  2. 2Build and hand-check the two-coordinate positional view

    Derive contiguous searchable-term positions while retaining complete scanner-token positions, then prove count, order, document, term, and source-index conservation.

  3. 3Analyze phrase and proximity queries

    Apply the compatible scanner, retain selected and ignored tokens, validate exact arity, and record invalid, missing, repeated, case-different, and non-ASCII cases.

  4. 4Find exact phrase matches with complete traces

    Intersect ordered positions under query offsets, retain overlapping and repeated-term matches, and show which candidate starts survive each step.

  5. 5Find bounded proximity matches with complete traces

    Apply after and either-side rules, calculate intervening terms, handle identical terms without self-pairs or duplicates, and retain boundary and rejection evidence.

  6. 6Build deterministic results, excerpts, and figures

    Apply the published evidence order, construct one source-linked excerpt per match, and save candidate-versus-match and proximity-gap figures from exact plot data.

  7. 7Serialize, reload, and replay the engine

    Save and read back the positional view, queries, traces, matches, excerpts, summaries, figures, configuration, identities, and manifest, then replay separately.

  8. 8Write and audit the engine card

    Explain searchable terms, both position systems, phrase adjacency, proximity limits, result order, excerpts, missing cases, artifacts, replay evidence, and lexical limits.

Required evidence

Save the versioned configuration and corpus, tokenizer, and TEXT-04 identities; positional_term_index.json; query_analysis.jsonl, phrase_matches.jsonl, and proximity_matches.jsonl; match_traces.jsonl, result_excerpts.jsonl, and query_summary.csv; candidate-versus-match and proximity-gap figures with their exact plot_data.json; phrase_proximity_manifest.json; artifact read-back checks; a replay agreement or first-mismatch record; and project-root report.md as the engine card.

Limits

This project does not teach arbitrary query syntax, Boolean operators, wildcards, regular expressions, spelling correction, synonyms, lowercasing, stemming, stop-word removal, unordered multi-term proximity, phrase or proximity scoring, relevance ranking, top_k, TF–IDF, BM25, probability, embeddings, vector search, query expansion, web crawling, access control, production indexing, or performance benchmarking. It does not claim that a lexical match proves relevance, meaning, popularity, fairness, or usefulness.

Review

The final review asks whether both position systems are conserved, query rules are explicit, phrase and proximity traces explain every result, identical-term pairs avoid self-matches and duplicates, result order is deterministic, excerpts preserve source tokens, replay is separate, and the engine card stays within the evidence.