Milestone 6 of 8

Save split and batch evidence

Write the grouped records, manifests, configuration, runtime evidence, and report, then reload what was saved.

The pipeline becomes reusable when its decisions survive the Python process. Save the source identity, movement choices, and invariant results together.

Goal

Write the required split and batch artifacts so another process can inspect the exact source, configuration, groups, batches, and omitted IDs.

Inputs

Use the aligned source boundary, validated configuration, split indexes, group records, batch result, generator implementation, and runtime information.

The required outputs are:

  • working_records.csv;
  • checking_records.csv;
  • split_manifest.json;
  • batch_manifest.json;
  • a saved configuration record with generator and runtime evidence;
  • a replay result and one controlled-comparison result;
  • a short evidence report.

Keep the exact source dataset and schema identity in the manifests. The split manifest should preserve ordered source positions and IDs for both groups. The batch manifest should preserve ordered batch positions and IDs and any omitted IDs. Record invariant results rather than only a final success label.

Deliverables

Implement the save step in src/main.py. Reload the CSV and JSON files after writing and record the source row count, group counts, batch counts, omitted IDs, configuration, generator implementation, compatible runtime, and conservation evidence in the report or the declared manifests.

Do not add a second source copy or silently replace an input artifact. Use stable column names and stable JSON fields so a later project can consume the valid boundary without guessing.

Checks

Check that every required file exists, opens as the expected format, and agrees with the source identity. Recompute group and batch counts from the saved files. Check that ordered IDs match positions, omitted IDs are disjoint from retained IDs, and the saved invariant results agree with recomputation.

Change one saved row or one manifest count in a temporary copy. The audit must report the disagreement and leave the original output untouched.

Workspace

Write generated files under output/. Keep the supplied input files and tests/public_cases.py read-only. The workspace should contain one continuous set of files shared by all milestones.

Hints

HintSave decisions, not just values
A list of IDs cannot explain how it was made. Include the source identity, configuration, ordered positions, generator, and invariant evidence.
HintReload what you wrote
An in-memory dictionary can hide a CSV or JSON serialization error. Close or bypass the original object before checking saved artifacts.

Review

Open each artifact as a reader who did not run the program. Can they tell which source was used, which IDs were held out, which IDs were omitted, and which configuration produced the result?

How to check your work

The supplied fixture writes only the required artifacts and keeps their fields tied to the accepted packet. Compare its evidence and failure behavior; do not copy a file merely because its internal grouping looks different.