Overflow and Underflow
Overflow happens when a number is too large to represent.
Underflow happens when a nonzero number is too close to zero to represent accurately.
Why Exponentials Are Risky
Exponentials grow quickly.
For large x, exp(x) can overflow.
For very negative x, exp(x) can underflow toward zero.
Both issues appear in softmax, likelihoods, and probability products.
For example, softmax uses exponentials. If one logit is very large, exp(logit)
may overflow before the final normalization can rescue the computation.
Likewise, multiplying many tiny probabilities can underflow toward zero even when the exact mathematical product is nonzero.
Underflow can be quiet. A calculation may not crash; it may simply replace a
small but meaningful value with 0. After that, taking a log, dividing by the
value, or comparing alternatives can give a completely different result.
Why Stable Rewrites Help
Stable formulas try to keep intermediate values inside a safe range.
The mathematical answer may be the same, but the path taken by the computer is safer.
That distinction is the heart of this chapter. Stable code often changes the intermediate quantities, not the mathematical quantity we intend to compute.
If exp(10000) is too large for the numeric format, is that overflow or
underflow?
Enter 1 for overflow, 2 for underflow.
Compute it first, then check your number.
Hint
Solution
It is overflow. Enter 1. The failure comes from a value being too large for
the numeric format, not too close to zero.
If a tiny nonzero probability is rounded down to 0, is that overflow or
underflow?
Answer it first, then check.
Hint
The value is too close to zero.
Solution
It is underflow because a tiny nonzero value became too small to represent accurately.
Which is more likely to overflow: exp(10) or exp(10000)?
Answer it first, then check.
Hint
Larger exponent means much larger output.
Solution
exp(10000) is much more likely to overflow. Exponentials grow very quickly,
so increasing the exponent from 10 to 10000 changes the scale enormously.
Do stable rewrites try to keep intermediate values in a safer numeric range?
Answer it first, then check.
Hint
The mathematical answer may be the same, but the computation path changes.
Solution
Yes. Stable rewrites compute equivalent quantities while avoiding unsafe intermediate scales. The intended mathematical value may be the same, but the computer follows a safer path to get there.
Enter 1 if a stable rewrite can preserve the intended mathematical quantity
while changing the intermediate values used by the computer.
Compute it first, then check your number.
Hint
Think about shifting before exponentiating.
Solution
Enter 1. A stable rewrite may compute the same mathematical quantity through
intermediate values that stay inside a safer numeric range.
If a tiny probability underflows to 0, can log(probability) become unsafe or
undefined in ordinary real-valued computation?
Answer it first, then check.
Hint
Ask what log(0) means.
Solution
Yes. In ordinary real-valued computation, log(0) is not a finite number. This
is one reason probability calculations often move into log space before tiny
products are formed.
Before Moving On
Exponentials are common in ML and need careful numerical handling.