Computation Graph Review

A neural network forward pass is a computation graph.

For one small unit:

z = wx + b
a = relu(z)
L = (a - y)^2

The graph records the order:

x, w, b -> z -> a -> L

Backpropagation follows the same graph backward:

L -> a -> z -> w, b, x

The backward pass does not invent new dependencies. It uses the dependencies created by the forward pass.

Exercise: Forward order

In z = wx + b, a = relu(z), L = (a - y)^2, which value is computed immediately after z? Enter 1 for a, or 0 for L.

Your answer

Compute it first, then check your number.

HintFollow the equations

a = relu(z) comes after z.

SolutionWork it out

The graph computes z, then a, then L.

Exercise: Backward direction

Backpropagation starts from the loss and moves backward. Enter 1 if this is true, or 0 if it starts from the input.

Your answer

Compute it first, then check your number.

HintStart at the final scalar

The loss is the final value whose gradients we want.

SolutionWork it out

Backpropagation starts with the loss and sends gradients backward through the graph.