Exploding and Vanishing Quantities
Repeated multiplication can make values grow or shrink quickly.
If a value is multiplied by 2 many times, it grows.
If a value is multiplied by 0.5 many times, it shrinks.
The important point is not only whether one factor is large or small. The important point is that the effect compounds.
For example:
but:
The two products use the same number of factors, but they move in opposite directions.
Gradients
The same idea appears in gradients.
During backpropagation, local derivatives are multiplied along paths.
If many factors are larger than 1, gradients may explode.
If many factors are smaller than 1, gradients may vanish.
The words "larger than 1" and "smaller than 1" are a useful first picture, not a full diagnosis. Real networks have many paths, matrices, nonlinearities, and normalization steps. Still, repeated products are the core reason scale can change so quickly.
ML Reading
Exploding and vanishing gradients make training unstable or slow.
Architectures, initialization, normalization, residual connections, and clipping can help control these issues.
Clipping can limit extreme gradients after they appear. Better initialization, normalization, and residual connections try to prevent harmful scale changes from building up in the first place.
So clipping is a guardrail, not a full explanation of stability. It can stop a single update from becoming too large, but it does not by itself fix why the gradient became extreme.
What is 0.5 * 0.5 * 0.5?
Compute it first, then check your number.
Hint
Solution
0.5 * 0.5 * 0.5 = 0.125. Each factor is below 1, so repeated
multiplication keeps shrinking the scale. This is the simple arithmetic pattern
behind vanishing quantities.
What is 2 * 2 * 2 * 2?
Compute it first, then check your number.
Hint
Multiply by 2 four times.
Solution
2 * 2 * 2 * 2 = 16. Each factor is above 1, so repeated multiplication
compounds growth instead of shrinkage.
If many local derivative factors are smaller than 1, do gradients tend to
explode or vanish?
Answer it first, then check.
Hint
Repeated multiplication by numbers below 1 shrinks scale.
Solution
They tend to vanish because repeated multiplication by factors below 1
shrinks the gradient.
Does gradient clipping prevent extreme gradients after they appear, or compute the exact same update as before?
Enter 1 for prevent extreme gradients, or 2 for exact same update.
Compute it first, then check your number.
Hint
The word clipping suggests limiting a value.
Solution
Gradient clipping limits extreme gradients after they appear. Enter 1. It
changes the update scale, but it does not by itself explain why the gradient
became extreme.
Enter 1 if gradient clipping limits extreme gradients after they appear but
does not by itself explain why they appeared.
Compute it first, then check your number.
Hint
Ask whether clipping changes the network path that produced the gradient.
Solution
Enter 1. Gradient clipping can limit an extreme gradient, but the cause may
still involve architecture, initialization, normalization, or data scale.
Before Moving On
Repeated products can change scale faster than intuition expects.