Backward Pass Overview

The backward pass asks how the loss changes with earlier values.

If:

L = (a - y)^2

then:

dL/da = 2(a - y)

This gradient moves backward to the operation that produced a.

If:

a = relu(z)

then:

dL/dz = dL/da * da/dz

That is the pattern of backpropagation:

upstream gradient x local derivative = downstream gradient

Here "downstream" means closer to the inputs and parameters.

Exercise: Loss derivative

Let L = (a - y)^2, with a = 5 and y = 2. What is dL/da?

Compute it first, then check your number.

HintUse the derivative

dL/da = 2(a - y).

SolutionWork it out

dL/da = 2(5 - 2) = 6.

Exercise: Multiply by local derivative

If dL/da = 6 and da/dz = 1, what is dL/dz?

Compute it first, then check your number.

HintChain rule

Multiply the upstream gradient by the local derivative.

SolutionWork it out

dL/dz = dL/da * da/dz = 6 x 1 = 6.