Project 12

Build a Ranked Retrieval and Evaluation Workbench

Compare an exact term-count ranker with one supplied tf-idf cosine ranker, trace every score, and evaluate both against complete binary judgments at fixed cutoffs.

  • 9 milestones
  • Optional
  • Browser workspace

Project question: Can you compare an exact term-count ranker with a supplied tf–idf cosine ranker, trace every score, and evaluate both against complete binary judgments at fixed cutoffs?

What makes this a project

Ranking is only one part of a retrieval experiment. In this project you will keep corpus, tokenizer, query, and index identity together; build sparse document and query weights; score the same candidate documents with two named systems; align complete judgments; and calculate supplied measures at fixed cutoffs. The saved contributions and rank traces let you inspect how every number was produced.

This workbench implements one fully specified experiment. The integer term-count baseline and the supplied tf–idf cosine rule are useful objects to compare, but a fixture result is not proof that either rule is best, relevant to future users, or meaningful outside the stated records.

You will produce:

  • checked corpus, tokenizer, index, query, judgment, and configuration identities;
  • sparse document frequencies, tf–idf weights, norms, and query vectors;
  • complete baseline and cosine score contributions and deterministic rankings;
  • rank-level judgment traces, missing-relevant evidence, and fixed-cutoff precision, recall, and F1 records;
  • paired candidate-minus-baseline differences and unweighted macro summaries;
  • comparable figures with exact plot data, serialized artifacts, read-back, and a separate replay; and
  • an evidence-limited retrieval-workbench card.

What you should know first

The project follows the Python path through Chapter 13. You should be able to validate and join records by stable IDs, count terms, use sparse mappings and NumPy arrays, calculate supplied weighted sums and norms, sort under exact ties, apply explicit zero-denominator rules, create comparable plots, save manifests, and replay a deterministic analysis. TEXT-04 and NUM-03 are useful but optional. If you skipped them, use the supplied version-matched fallback; do not invent a tokenizer, index, scorer, or judgment rule.

Supplied inputs

Use accepted documents, complete token records, corpus and tokenizer identity, and the TEXT-04 positional inverted index or version-matched fallback. Also use plans/queries.jsonl, plans/relevance_judgments.csv, plans/retrieval_config.json, and the tiny hand-check fixtures. The workbench compares exactly term_count_baseline and tfidf_cosine, using exact searchable token text, corpus order, query order, document IDs, and tokenizer identity.

The configuration supplies system identities, formula versions, the natural-log rule, score rounding, positive cutoffs, ranking ties, plot order, and numerical tolerances. Judgments contain one exact binary value for every configured query/document pair. This complete fixture is a local contract, not a claim that ordinary search evaluation has complete judgments.

The project workspace

project/
  README.md
  data/                              # supplied, read only
  plans/queries.jsonl                # supplied, read only
  plans/relevance_judgments.csv      # supplied, read only
  plans/retrieval_config.json        # supplied starting config
  src/config.py                      # reader implementation
  src/weights.py                     # reader implementation
  src/queries.py                     # reader implementation
  src/score.py                       # reader implementation
  src/judgments.py                   # reader implementation
  src/measures.py                    # reader implementation
  src/compare.py                     # reader implementation
  src/report.py                      # reader implementation
  src/main.py                        # reader implementation
  output/                            # generated artifacts
  tests/public_cases.py              # supplied, read only

Keep one continuous workspace through every milestone. Weight construction, query analysis, scoring and ranking, judgment alignment, measure calculation, comparison, and reporting should remain independently inspectable.

Project milestones

Work through these in order. Each milestone produces evidence used by the next one, while project completion remains separate from lesson progress.

  1. 1Verify corpus, tokenizer, index, queries, judgments, and configuration

    Check source identities, complete query/document judgment coverage, token rules, formulas, system names, cutoffs, rounding, tolerances, and zero-term documents.

  2. 2Calculate document frequencies, weights, and norms

    Build sparse non-zero document weights, retain zero-term documents, and prove term-count, posting, smoothed natural-log idf, and norm invariants.

  3. 3Analyze queries and build sparse query vectors

    Apply compatible token rules, preserve repeated, ignored, and missing terms, reuse corpus idf, and represent zero-norm queries explicitly.

  4. 4Score and rank both systems with complete traces

    Reproduce the integer baseline, calculate tf-idf cosine contributions and norms, apply exact rounding and ties, and retain complete positive rankings.

  5. 5Align judgments and build rank-level relevance traces

    Join by stable query/document identity, record cumulative relevant hits, reject duplicates, and retain relevant documents missing from each ranking.

  6. 6Calculate fixed-cutoff retrieval measures

    Calculate precision, recall, and F1 at every cutoff while retaining actual prefix sizes, hits, denominators, and all zero-result rules.

  7. 7Compare systems and create fair figures

    Align per-query measures, calculate candidate-minus-baseline differences and unweighted macro means, and plot both systems with shared orders and scales.

  8. 8Serialize, reload, and replay the workbench

    Save and read back weights, queries, scores, rankings, judgments, measures, comparisons, figures, configuration, identities, and manifest, then replay separately.

  9. 9Write and audit the retrieval-workbench card

    State scoring, rounding, tie, judgment, cutoff, denominator, comparison, artifact, replay, and interpretation boundaries from the saved evidence.

Required evidence

Save versioned retrieval configuration and corpus, tokenizer, index, and query identity; term_statistics.csv, document_weights.csv, document_norms.csv, query_analysis.jsonl, and query_weights.jsonl; score_contributions.csv and complete ranked_results.jsonl for both systems; rank_relevance.csv and missing_relevant_documents.jsonl; retrieval_measures.csv, paired_measure_differences.csv, and macro_measure_summary.csv with explicit denominators; comparable per-cutoff and paired-difference figures with exact plot_data.json; retrieval_workbench_manifest.json, artifact read-back checks, a separate replay agreement or first-mismatch record; and project-root report.md.

Limits

This project does not teach phrase or proximity matching, Boolean syntax, stemming, stop-word policy, spelling correction, BM25, probabilistic retrieval, query expansion, learned embeddings, vector databases, approximate search, graded or incomplete judgments, judgment pooling, assessor agreement, average precision, MAP, MRR, R-precision, NDCG, interpolated curves, statistical significance, confidence intervals, model selection, fairness or causal evaluation, click feedback, online experiments, web crawling, production indexing, performance benchmarking, or retrieval-augmented generation. It does not claim that a higher score or supplied-fixture measure proves general relevance or usefulness.

Review

The final review asks whether identities and token rules remain compatible, zero-term records remain visible, every score is traceable, ties and cutoffs are deterministic, judgment coverage is complete, denominators are explicit, macro-means are unweighted, plots use shared order and scales, replay is separate, and the card rejects semantic, causal, significance, fairness, representative, and future-performance claims.