Numerical Stability in Training
Training combines optimization with numerical computation.
A training loop may fail because the idea is wrong, but it may also fail because the numbers become unstable.
Common Symptoms
Watch for:
- loss becomes
NaN - gradients explode
- activations become too large
- probabilities become exactly
0 - updates are much larger than parameters
These symptoms are clues, not final diagnoses. A NaN loss might come from an
unsafe logarithm, overflow, division by zero, an excessive learning rate, or a
bug in the model code.
Practical Checks
Useful checks include:
- inspect loss values
- inspect gradient norms
- reduce learning rate
- check stable softmax and log-sum-exp
- run a small gradient check
The goal is to narrow the failure. If the loss becomes NaN after the first
update, inspect the forward pass, loss computation, gradients, and update size
in that order.
Numerical stability is not separate from optimization. A learning rate can turn a reasonable gradient into a destructive update.
A useful debugging habit is to compare scales. If the update is larger than the parameter, or the gradient norm suddenly jumps by orders of magnitude, the training loop is telling you where to look.
If training loss becomes NaN, is that a possible numerical stability symptom?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
NaN means not a number.Solution
Yes. NaN loss is a common numerical failure symptom. Enter 1. It means the
training loop produced a value that is no longer a usable number.
If updates are much larger than the parameters themselves, is that a stability warning sign?
Answer it first, then check.
Hint
The symptom list mentions update size.
Solution
Yes. Updates much larger than the parameters can indicate unstable training. That scale mismatch suggests the optimizer may be changing the model too violently.
If training diverges because the learning rate is too large, should reducing the learning rate often be a useful check?
Answer it first, then check.
Hint
The practical checks include reducing the learning rate.
Solution
Yes. Reducing the learning rate is often a useful check when updates are too large or training diverges.
Is NaN loss by itself a complete diagnosis of the cause?
Answer it first, then check.
Hint
The lesson says symptoms are clues, not final diagnoses.
Solution
No. NaN loss is a symptom. The cause might be overflow, unsafe logarithms,
division by zero, excessive learning rate, or a bug.
Enter 1 if comparing update size with parameter size can help diagnose
unstable training.
Compute it first, then check your number.
Hint
Think about what happens if one update is much larger than the parameter itself.
Solution
Enter 1. If updates are much larger than the parameters, optimization may be
changing the model too violently.
Before Moving On
Optimization is not separate from numerical stability in real training.