Train a line from first principles

A model small enough to inspect

Imagine measuring a quantity from a single input. We will use five synthetic pairs to keep the arithmetic visible. Our prediction is ŷ = wx + b: w controls the slope and b the intercept. The hat on ŷ means a prediction; y is the target.

These points follow y = 2x + 1 exactly. Knowing that answer lets us check the training implementation; our algorithm will start at w = b = 0 and use the training pairs to reduce its error.

This is regression: predicting a number. A classifier predicts a category. Logistic regression would be a separate classification project.

This exact five-point line is a hand-checkable warm-up. The project workspace uses ten slightly noisy observations, with seven fixed for training and three reserved for evaluation.

1. Measure how wrong the line is

For each of the n training pairs, subtract the target from the prediction, square that error, and average the results. This is mean squared error, or MSE.

L = (1/n) Σᵢ (wxᵢ + b − yᵢ)²

At w = b = 0, the predictions are all zero. The squared errors are 9, 1, 1, 9, and 25. Their mean is 9. Reproduce this value before implementing training.

2. Turn error into an update

A gradient tells us how the loss changes as a parameter changes. For this loss, each error contributes to two derivatives:

∂L/∂w = (2/n) Σᵢ (wxᵢ + b − yᵢ)xᵢ
∂L/∂b = (2/n) Σᵢ (wxᵢ + b − yᵢ)

The factor 2 comes from differentiating the square. The extra xᵢ in the first expression comes from differentiating wxᵢ with respect to w. Subtract a small multiple of each derivative to move downhill.

dw, db = gradients(TRAIN, w, b)
w, b = w - learning_rate * dw, b - learning_rate * db

Both derivatives use the same old w and b. At the starting point, dw = −8 and db = −2. With learning rate 0.1, the first update gives w = 0.8 and b = 0.2. Predict whether the training error will decrease, then take one step.

3. Build your experiment

Use the continuous workspace below. Begin with the data loader, then implement the model, loss, gradients, optimization, evaluation, and report in order. The evaluation rows must stay out of every parameter update.

Each milestone has a small public check. Passing it confirms the stated interface and deterministic examples; it does not judge the quality of your explanation or create an authoritative submission.

4. Compare, then question the result

Fit the constant baseline from the seven training targets, then measure both the baseline and your line on the three reserved rows. A model should improve on a simple alternative, not merely produce a small-looking number.

A least-squares fit of the fixed training split is approximatelyw = 1.4885 and b = 0.7830. Its evaluation MSE is about 0.0161; the training-mean baseline is about25.0056. Use these values to diagnose the complete run, not as replacements for the milestone checks.

The reserved rows come from the same small synthetic relationship as training. They check plumbing and basic generalization, but they do not establish robustness to a new population. Repeatedly choosing settings from their result would also compromise their independence.

Completion evidence

Complete the seven milestones and keep the versioned files in the workspace. Your report.md should identify the split, parameter order, gradient comparison, optimization settings, fitted values, baseline comparison, conclusion, limitations, and reproduction steps. Explain why this single affine model needs no ReLU.