Align judgments and build rank-level relevance traces

An ordered result is not yet an evaluation. Join each returned document to the complete binary judgments, while keeping relevant documents that were never returned visible in the recall denominator.

Goal

Align both complete rankings to judgments by stable query/document identity and record rank-level evidence, cumulative unique hits, total relevant documents, and missing relevant documents.

Inputs

Use complete plans/relevance_judgments.csv, both ranked-result streams, query and document manifests, and validated identity rules. Join only on (query_id, document_id), never on row position. A generated ranking must not contain duplicate document IDs for one query and system.

For every ranked document, retain rank, document ID, score, binary judgment, cumulative unique relevant hits, and the total number of relevant corpus documents for that query. Relevant documents absent from a ranking remain in the recall denominator and receive a separate missing-relevant record.

Deliverables

Implement src/judgments.py. Under output, write rank_relevance.csv with one row per returned document for each system. Also write missing_relevant_documents.jsonl, naming each query, system, and relevant document not retrieved. Include query order, system identity, rank, judgment, cumulative hit count, and total-relevant denominator.

Checks

Reject missing, duplicate, extra, or unknown judgment identities even if the row count looks correct. Reject duplicate ranked document IDs within a query. Check shuffled judgment rows, queries with zero, one, and several relevant documents, documents not retrieved, complete rank coverage, rank ordinals starting at one, and cumulative hits that increase only on a first retrieved relevant document.

Check every ranked document joins exactly once, every total-relevant count comes from the complete judgment table, and missing relevant documents are not silently discarded. Verify the two systems use the same judgment records and that inputs remain unchanged.

Workspace

Keep identity joining and rank-level trace construction in src/judgments.py. Read rankings and judgments without mutation. Do not calculate precision, recall, F1, comparison differences, or figures yet.

Hints

Review

Pick a query with one relevant document that was not retrieved. Where should it appear in the rank trace, and where should it still affect later recall?

How to check your work

Checks compare rank traces and missing-relevant records with the deliberately complete and deliberately broken fixtures. The supplied fixture joins by stable IDs and keeps unretrieved relevant documents visible.