Calculate the quality summary

The three files now show where every row went. This milestone reduces the classified data to a small set of descriptive counts without hiding the rows that produced them.

Milestone goal

Calculate a JSON quality summary from the classified outputs. Report status and reason counts, then summarize finite readings for each sensor using valid rows only.

Summary contract

Implement an interface such as:

Write output/quality_summary.json with this shape:

{
  "source_row_count": 0,
  "status_counts": {
    "valid": 0,
    "missing": 0,
    "rejected": 0
  },
  "reason_counts": {},
  "valid_readings": {
    "count": 0,
    "by_sensor": {
      "north": {"count": 0, "minimum": null, "maximum": null, "mean": null},
      "south": {"count": 0, "minimum": null, "maximum": null, "mean": null}
    }
  }
}

source_row_count is the number of data rows, not the header. The three status keys always appear, even when one count is zero. reason_counts contains the stable reasons for non-valid rows. For each sensor, calculate count, minimum, maximum, and arithmetic mean from valid finite readings only. When a sensor has no valid readings, use count 0 and JSON null for all three numerical fields; do not use 0 to suggest a measurement.

The summary describes the supplied rows. It is not an estimate of sensor behavior outside this file.

Deliverables

Save:

• output/quality_summary.json;
• a short calculation note in report.md showing one per-sensor count and mean from the classified rows;
• a read-back check that the JSON agrees with the CSV partitions.

Checks

Use a hand-checkable fixture to verify:

• status counts add to source_row_count;
• reason counts add to the number of non-valid rows;
• valid readings are the only values used for minimum, maximum, and mean;
• north and south remain separate groups;
• an empty valid group uses the stated 0/null policy;
• the summary survives JSON write and read-back without changing types or names;
• the original classified records remain unchanged.

Calculate one mean by hand before trusting the program. A mean that looks reasonable can still include a rejected row or silently treat a blank as zero.

Review

Read reason_counts beside rejected_records.csv. Every count should be traceable to rows with the same reason. Read valid_readings.by_sensor beside valid_records.csv; do not infer a cause for a high or low value.

Next step

The visual report will show these counts and valid readings. It must use the same underlying records and keep the plot descriptive rather than decorative.

How to check your work

The supplied fixture includes the exact JSON shape and the empty-group policy. Compare counts and source rows before comparing implementation details.