Review

Key ideas

• Backpropagation sends gradients backward from the loss.
• It uses the same graph created by the forward pass.
• Each operation contributes a local derivative.
• For z = wx + b, dL/dw = (dL/dz) x.
• For z = wx + b, dL/db = dL/dz.
• Bias gradients accumulate across batch examples.
• ReLU passes gradients when active and blocks them when inactive.
• Gradient checking compares backpropagation with finite differences.

Common formulas

dL/dz = dL/da * da/dz
dL/dw = dL/dz * dz/dw = g x
dL/db = dL/dz * dz/db = g

Common mistakes

• Forgetting that ReLU's backward behavior depends on the forward value.
• Forgetting to accumulate gradients across examples.
• Treating gradient checking as a replacement for backpropagation.
• Confusing the gradient with the parameter update. Updates come next.

Before moving on

You should be able to compute a scalar weight gradient, a scalar bias gradient, a ReLU backward step, and a finite-difference gradient check.