Validate configuration and match every item by ID

Start with identity. A score is not meaningful when an expected label has been paired with the wrong prediction.

Goal

Validate the configuration and match every item by item_id, preserving expected, baseline, candidate, and source-group fields in one auditable record.

Inputs

Use the read-only data/items.csv, data/baseline_predictions.csv, and data/candidate_predictions.csv, plus plans/evaluation_config.json. The configuration supplies the ordered allowed labels, display names, and source-group field.

The item file must have unique item_id, an expected label from the configured order, and a non-empty source group. Each prediction file must contain exactly the item IDs in the item file, one prediction per ID, and labels from the same configured order. The candidate file is intentionally in a different row order.

Deliverables

Implement src/config.py and src/data.py to validate the schema and build output/aligned_items.csv. Preserve source row position and include item ID, expected label, both predictions, source group, and both match statuses.

Record the configuration identity and input counts in the report or a stable manifest section. Do not align rows by position and do not silently repair a duplicate, missing, extra, or unknown-label record.

Checks

Reject duplicate IDs, missing IDs, extra prediction IDs, unknown labels, missing fields, empty source groups, and invalid configuration labels. Shuffle the candidate prediction rows and prove that the aligned result is unchanged.

Check that every item appears exactly once, all required fields are preserved, both match statuses agree with the labels, and the supplied files and arrays are not mutated. Use public fixtures for each identity failure.

Workspace

Keep configuration validation in src/config.py and file loading and ID joins in src/data.py. Leave all supplied files read-only. Write the aligned table under output/ and do not calculate aggregate measures before the join is validated.

Hints

Review

Choose an item near the end of the item file and follow it into the aligned table. Explain why stable identity is more reliable than matching two files by their current row order.

How to check your work

Checks compare the aligned schema and invalid-input behavior with the supplied fixture. The supplied fixture preserves every item rather than hiding a failed join.