Saturation
An activation saturates when large changes in input produce only tiny changes in output.
Sigmoid is the easiest example. For large positive inputs, sigmoid gets very close to 1. For large negative inputs, it gets very close to 0.
sigmoid(10) is close to 1
sigmoid(-10) is close to 0
Near those flat regions, the function changes slowly.
That matters because training depends on gradients. If an activation is nearly flat, its local derivative is small. Small local derivatives can make earlier parameters receive weak learning signals.
ReLU saturation is different
ReLU does not saturate on the positive side:
relu(10) = 10
relu(50) = 50
But ReLU is flat for negative inputs:
relu(-10) = 0
relu(-50) = 0
So ReLU can still produce inactive units for some inputs.
If an activation is almost flat for large positive inputs, enter 1 if this can make gradients small, or 0 if it cannot affect gradients.
Compute it first, then check your number.
HintGradient means local change
A flat curve changes slowly.
SolutionWork it out
A nearly flat activation has a small local derivative. During backpropagation, that small derivative can weaken the gradient signal.
What is relu(-8)?
Compute it first, then check your number.
HintUse max
Compute max(0, -8).
SolutionWork it out
relu(-8) = max(0, -8) = 0.