Inspect and Validate a Dataset

Project question: Can we turn a small measurement file into a dataset that another reader can inspect, check, and use without hiding missing or rejected rows?

What you will build

You will build the first durable data boundary in the Python project ladder. Starting with one supplied CSV file, you will preserve the source rows, apply a fixed schema, classify every row, write separate outputs, calculate a quality summary, make a small visual report, and save enough evidence for another run to audit the result.

This project is about careful data handling. It does not teach imputation, database design, streaming, or a general theory of data quality. A missing reading stays missing. A row that fails the contract stays visible as rejected evidence.

The supplied file

The read-only file data/raw_measurements.csv contains these fields:

The file contains ordinary valid rows, a blank reading, a repeated identity, an invalid numeric value, an unknown sensor, and a unit mismatch. These are small deliberate fixtures. They make the decisions inspectable; they are not a list of every defect that a real dataset can contain.

The supplied src/schema.py owns the field names and schema revision. Do not edit the raw file or silently change the schema while working through the milestones.

The classification rule

Every source data row enters exactly one output:

1. valid — the complete row satisfies the schema and may enter numerical work;
2. missing — the identity and every non-reading field are valid, but the reading is blank;
3. rejected — the row cannot safely enter the dataset because an earlier identity, sensor, observation, unit, parsing, or range rule failed.

Apply checks in this order: row and column structure, non-empty first-seen record_id, sensor, observation, unit, and reading. A later duplicate ID is rejected while the first occurrence continues through the remaining checks. A blank reading is missing only when every preceding rule passes. If several rules fail, keep the first failure and its stable machine-readable reason.

The project never fills a value, silently removes a duplicate, or treats all excluded rows as one kind of failure.

What you should know first

The project follows Python Chapters 1–12. You should be able to organize a small multi-file program, read delimited text, use records and arrays, check inputs, calculate summaries, make a Matplotlib figure, and save JSON or CSV artifacts. Chapter 13 randomness is not required.

You do not need to remember every detail before beginning. Return to the relevant lesson when a milestone exposes a gap. The project is optional and does not silently mark lessons or the chapter complete.

The project workspace

project/
  README.md
  data/raw_measurements.csv        # supplied, read only
  src/schema.py                    # supplied field names and constants
  src/pipeline.py                  # your parsing and classification code
  src/report.py                    # your summary and report code
  src/main.py                      # your orchestration code
  output/                          # generated artifacts
  tests/public_cases.py            # supplied, read only

Keep parsing, classification, reporting, and orchestration separable and testable. The file split may be simplified during implementation review when that makes the contracts clearer; do not add modules only to make the tree look larger.

Each milestone changes the evidence available to the next one. Reading a milestone is not the same as completing it. A milestone is ready for completion when its stated artifacts exist, its automated checks pass, and you have read the evidence behind those checks.

Final artifacts

The completed project contains:

• valid_records.csv;
• missing_records.csv;
• rejected_records.csv, including source line and reason;
• quality_summary.json with status, reason, and per-sensor counts;
• quality_report.png;
• dataset_manifest.json;
• a short findings section;
• a saved audit showing that the artifacts agree.

The next data project consumes only the valid records and the manifest, while the missing and rejected files remain available as evidence about what was excluded.

How the result should be described

The summary describes this supplied file. It does not establish a general quality rule for sensor data, explain why a defect occurred, or justify automatically repairing it. Keep observations separate from guesses about causes. A reader should be able to trace every count to a classified row and every plotted point to a source identity.

Sources and reuse

The project uses the supplied measurement fixture and schema. Its reusable boundaries are the parser, stable row-status and reason vocabulary, classified CSV columns, quality summary, and manifest. Later Projects may use those boundaries without assuming that this project was completed.