Chain Rule Through Layers

A layer is a group of operations. Backpropagation still uses local derivatives.

For:

z = wx + b
a = relu(z)
L = (a - y)^2

the backward chain is:

dL/da
dL/dz = dL/da * da/dz
dL/dw = dL/dz * dz/dw
dL/db = dL/dz * dz/db

The layer does not need a special kind of calculus. It needs careful bookkeeping.

xu = 2xL = u^2du/dx = 2dL/du = 2udL/dx = dL/du * du/dx
Backpropagation multiplies upstream gradients by local derivatives.

Reusing upstream gradients

Once we know dL/dz, it becomes the upstream gradient for every input to z = wx + b.

That same value helps compute gradients for x, w, and b.

Exercise: Chain through activation

If dL/da = 4 and da/dz = 0, what is dL/dz?

Compute it first, then check your number.

HintMultiply

Chain rule multiplies the two values.

SolutionWork it out

dL/dz = 4 x 0 = 0.

Exercise: Chain through linear part

If dL/dz = 5 and dz/dw = 3, what is dL/dw?

Compute it first, then check your number.

HintUpstream times local

Use dL/dw = dL/dz * dz/dw.

SolutionWork it out

dL/dw = 5 x 3 = 15.