Milestone 6 of 9

Calculate fixed-cutoff retrieval measures

Calculate precision, recall, and F1 at every cutoff while retaining actual prefix sizes, hits, denominators, and all zero-result rules.

Cutoffs turn a complete ranking into a small, explicit measurement record. Keep the actual prefix and every denominator visible, including zero cases.

Goal

Calculate the supplied precision, recall, and F1 measures for every system, query, and configured cutoff, using the rank-level judgment traces.

Inputs

Use rank_relevance.csv, configured unique positive non-boolean integer cutoffs, and query/system order. For cutoff k, inspect the first min(k, returned_count) ranks. Let hits be the number of relevant documents in that prefix:

precision = hits / retrieved_positions      # 0.0 when none were retrieved
recall = hits / total_relevant_documents     # 0.0 when none are relevant
f1 = 2 * precision * recall / (precision + recall)
     # 0.0 when precision + recall is zero

A cutoff larger than the returned ranking uses every returned document; it does not add imaginary positions. These are supplied operational definitions for this workbench.

Deliverables

Implement src/measures.py. Produce output/retrieval_measures.csv with one row per system, query, and cutoff, including cutoff, actual retrieved positions, hits, every denominator, precision, recall, and F1. Preserve query, system, and cutoff display order.

Checks

Hand-check empty rankings, rankings shorter than, equal to, and longer than a cutoff, queries with zero, one, and several relevant documents, and every zero-denominator rule. Check that actual prefix size is min(k, returned_count), hits count relevant ranks only once, precision is zero when no positions are retrieved, recall is zero when no documents are relevant, and F1 is zero when precision plus recall is zero.

Check all three measures against rank traces, explicit denominators, configured cutoff order, deterministic CSV output, and no imaginary zero-score results. Do not substitute a different measure or denominator policy.

Workspace

Keep cutoff prefix selection and measure calculation in src/measures.py. Read rank traces and configuration without mutation. Do not calculate paired differences or figures yet.

Hints

HintName the prefix

Store both configured cutoff and actual retrieved_positions. They differ when a ranking is shorter than the cutoff.

HintCarry the denominator

Read total_relevant_documents from the complete judgment evidence, not from the prefix. Recall can remain low when a relevant document was never retrieved.

HintGuard the final division

Compute precision and recall with their stated zero cases, then handle a zero precision-plus-recall sum before dividing for F1.

Review

Use a short ranking and a cutoff larger than it. Which positions count, what are the precision and recall denominators, and why must the extra positions not be invented?

How to check your work

Checks compare every cutoff row with the hand-checked rank traces. The supplied fixture applies only the supplied formulas and preserves actual prefix lengths and denominators.