Loop Anatomy

A training loop repeats the same small pattern:

  1. Get a batch.
  2. Compute predictions.
  3. Compute loss.
  4. Compute gradients.
  5. Update parameters.
  6. Record useful numbers.

The model does not improve during the forward pass alone. The forward pass only produces predictions. The loss measures error. The backward pass computes gradients. The optimizer uses those gradients to change the parameters.

For one scalar parameter, the shape of the update is:

w_next = w - learning_rate * dL/dw

The minus sign matters. The gradient points toward increasing loss. Training usually wants to move in the opposite direction.

current wnext wsubtract eta gradientw_next = w - eta * dL/dw
An optimizer turns a gradient and a step size into a parameter change.

The loop is simple, but the details matter. A bad learning rate, a broken gradient, a shuffled label, or an unstable loss can make the loop fail even when the model architecture is reasonable.

Exercise: Name the missing step

A loop computes predictions and loss, then immediately changes parameters. Which step is missing between loss and update? Enter 1 for backward pass, 2 for validation, or 3 for logging.

Compute it first, then check your number.

Exercise: One scalar update

Let w = 5, dL/dw = 2, and learning_rate = 0.1. What is the next value of w?

Compute it first, then check your number.