Backprop Through ReLU
ReLU is:
relu(z) = max(0, z)
For backpropagation, its local derivative is simple:
da/dz = 1 when z > 0
da/dz = 0 when z < 0
At exactly z = 0, implementations choose a convention. For this course, treat the derivative at zero as 0 unless stated otherwise.
If the unit was active in the forward pass, the gradient passes through. If it was inactive, the gradient is blocked.
Example
If:
z = -2
dL/da = 7
then:
da/dz = 0
dL/dz = 7 x 0 = 0
Exercise: Active ReLU gradient
If z = 3 and dL/da = 6, what is dL/dz through ReLU?
Compute it first, then check your number.
HintPositive pre-activation
Active ReLU passes the gradient through.
SolutionWork it out
Since z = 3 > 0, da/dz = 1, so dL/dz = 6 x 1 = 6.
Exercise: Inactive ReLU gradient
If z = -2 and dL/da = 6, what is dL/dz through ReLU?
Compute it first, then check your number.
HintNegative pre-activation
Inactive ReLU blocks the gradient.
SolutionWork it out
Since z = -2 < 0, da/dz = 0, so dL/dz = 6 x 0 = 0.