Project 2
Build a Reproducible Split and Batch Pipeline
Move aligned records into a checking group and working batches while preserving every row's identity and the evidence needed to replay the run.
Project question: Can you move checked records into a reproducible checking group and working batches without losing the identity or alignment of any row?
What makes this a project
Splitting a table and making batches can look like a few indexing operations. The important work is keeping the evidence attached to the right record. This project makes that boundary visible: one permutation chooses positions, every field follows the same positions, and each decision is saved so another run can be compared with it.
You will build a small system that produces:
- working and checking records with their original IDs and fields;
- a manifest for the source, configuration, ordered indexes, and groups;
- batches made from the working positions, including any omitted remainder;
- replay evidence for the same settings;
- one controlled comparison in which only one setting changes;
- a short report that explains what stayed aligned and what changed.
The project does not decide whether a split is suitable for training, testing, validation, or a scientific claim. It teaches a dependable movement of records. Later subjects can give that movement a domain-specific meaning.
What you should know first
The project follows the Python path through Chapter 13. You should be able to
read CSV and JSON files, use functions and records, work with lists and NumPy
arrays, check shapes and values, preserve inputs, and create one explicit
NumPy generator. DATA-01 is useful preparation, but it is optional: a supplied
version-matched valid_records.csv and dataset_manifest.json provide the
same starting boundary when you did not complete it.
The supplied boundary
The input has four aligned columns:
record_id sensor observation reading
The first axis is the record axis. A value at position i in one column refers
to the same source record as position i in every other column. Keep the
source order and IDs visible before any permutation. Do not rename these
columns as features, targets, labels, or tokens; later subjects may choose
those meanings for their own work.
Configuration and movement
One configuration contains a seed, an exact checking_count, a positive
batch_size, and a keep_remainder choice. The pipeline validates the
configuration first, creates one permutation of source positions, sends the
first positions to the checking group, and sends the rest to the working
group. It then partitions the working positions in their current order.
Project milestones
Work through these in order. Each milestone produces evidence used by the next one, while project completion remains separate from lesson progress.
- 1Inspect and verify aligned fields
Load the supplied records and prove that each ID, sensor, observation, and reading still describes the same source row.
- 2Define and validate the configuration
Name the seed, exact checking count, batch size, and remainder rule, then reject impossible values before random work begins.
- 3Create a split from one permutation
Use one passed generator and one permutation to produce disjoint checking and working index arrays.
- 4Select groups through shared indexes
Apply each index array to every field and verify coverage, uniqueness, disjointness, and row alignment.
- 5Form working batches
Partition the working order into bounded batches and make every retained or omitted remainder ID visible.
- 6Save split and batch evidence
Write the grouped records, manifests, configuration, runtime evidence, and report, then reload what was saved.
- 7Replay the complete pipeline
Rebuild the run from source files and saved evidence and report exact agreement within the recorded compatibility boundary.
- 8Make one controlled comparison
Change one configuration field, predict the effect, and verify that all unchanged alignment and conservation rules still hold.
How completion works
Project progress is separate from Python chapter and lesson progress. A milestone is ready to mark done when its files, saved evidence, and required checks are complete. Passing checks does not by itself prove that the report explains alignment clearly; read the output and compare the source IDs yourself.
The reusable boundary is small: configuration, one-index selection, aligned field movement, batch membership, split and batch manifests, and replay fixtures. A later project may use the boundary for images, tokens, labels, or measurements without inheriting this project's domain meaning.
Limits
This project makes one pass of batches. It does not introduce training, validation, or test semantics; stratification; group- or time-aware splitting; epochs; reshuffling; data-loader frameworks; parallel workers; probability claims; or performance comparisons. Exact random replay is expected only for a compatible recorded generator and runtime, not for every future library version.
Review
The final review asks whether every source index appears exactly once, whether all fields remain paired with their IDs, whether the remainder policy is visible, whether the saved run can be replayed, and whether the comparison changes only the declared setting. The report should state what the evidence supports and what it does not.
The project may later supply a checked splitter and canonical manifest to a reader who skipped it. That fallback teaches the later subject's own split policy rather than pretending that this project established one.