Forward, Loss, Backward, Update

The core training step has four verbs.

Forward computes predictions:

y_hat = model(x)

Loss compares predictions with targets:

L = loss(y_hat, y)

Backward computes gradients:

dL/dw, dL/db, ...

Update changes parameters:

w = w - learning_rate * dL/dw
b = b - learning_rate * dL/db

These verbs should stay separate in your mind. A common beginner mistake is to treat the model call as if learning happens inside it. The model call only computes. Learning happens when parameters are updated.

For a one-parameter model y_hat = wx, with squared loss L = (y_hat - y)^2, the full step is visible:

y_hat = wx
L = (y_hat - y)^2
dL/dw = 2(y_hat - y)x
w_next = w - learning_rate * dL/dw

Exercise: Loss from prediction

Let y_hat = 7 and y = 4. For L = (y_hat - y)^2, what is the loss?

Your answer

Compute it first, then check your number.

Exercise: Gradient for one weight

Let y_hat = 7, y = 4, and x = 2. For L = (y_hat - y)^2 and y_hat = wx, what is dL/dw?

Your answer

Compute it first, then check your number.