Write the three classified datasets

Classification is useful only if it can be saved without losing the source trace. This milestone turns the in-memory records into three stable files and proves that the partitioning did not drop or duplicate a row.

Milestone goal

Write valid_records.csv, missing_records.csv, and rejected_records.csv with stable columns, preserved identities, and explicit reasons.

Inputs and output schema

Use the classified records from the previous milestone. All three files must use this column order:

source_line,record_id,sensor,observation,reading_raw,reading,unit,
status,reason,reason_detail

The reading_raw field preserves the source text. reading is the parsed finite number for a valid row and is blank for missing or rejected rows. The status column agrees with the filename. reason is blank for valid rows and uses the stable reason vocabulary for the other two files. reason_detail is short reader-facing evidence, not a guessed cause.

Keep source order inside each output. Quote CSV fields correctly, including commas or quotes if an alternate fixture contains them. Never overwrite the raw source file.

Write one partition function

Implement an interface such as:

Write to a temporary location or otherwise avoid leaving a half-written final file when one output fails. A clear exception is better than three files that look complete but disagree.

Conservation contract

For the source data rows, prove all of the following:

• every row appears in exactly one output;
• no output contains a duplicate source_line or record_id unless the source itself contains the later duplicate, which must remain a rejected row with its own source line;
• concatenating the three output identities in source order recovers the complete source identity list;
• every output row retains the source line and raw field values;
• valid rows alone contain parsed readings suitable for numerical work.

The second condition distinguishes a source duplicate from accidentally writing one output row twice. Do not deduplicate the evidence while saving it.

Deliverables

Save:

• output/valid_records.csv;
• output/missing_records.csv;
• output/rejected_records.csv;
• a conservation result in the milestone report or output/partition_audit.json.

The audit should include source row count, each output count, status counts, and whether source lines and IDs were conserved.

Checks

Run the writer against:

• the public fixture containing all three statuses;
• an all-valid fixture;
• a fixture with no missing rows;
• a fixture with several rejected rows and a repeated source ID.

Read the files back and check exact headers, row counts, status values, stable ordering, raw values, and conservation. A check that only counts rows is not enough: two rows with the same shape can still carry different identities.

Review

Open one row from each file and trace it to the parser and source line. Check especially that a blank reading remains blank in the missing file and an invalid reading remains visible as raw text in the rejected file.

Next step

The quality summary will read these classified outputs and calculate facts about valid readings. It must not calculate a mean from missing or rejected rows.

How to check your work

Checks compare the reference writer's files and audit with your read-back result. The important evidence is the preserved boundary, not the choice of CSV helper.