LLM Primer

Overflow and Underflow

Overflow happens when a value is too large to represent.

Underflow happens when a nonzero value is too small to represent and collapses toward zero.

representable rangeunderflowoverflowtoo close to 0too large
Underflow pushes tiny values toward zero. Overflow exceeds the largest representable scale.

Deep learning can create both:

  • exponentials of large logits can overflow
  • products of small probabilities can underflow
  • gradients can become too large or too tiny
  • repeated operations can amplify scale problems

For example:

exp(1000)

is not a harmless large number in ordinary floating-point code. It can overflow.

Numerical stability often means rewriting the computation so the mathematical intent remains but the intermediate values stay representable.

DL-C16-T03-001Exercise: Overflow clue

Enter 1 for overflow warning, or 2 for normal small value: computing exp(1000).

Compute it first, then check your number.

DL-C16-T03-002Exercise: Underflow direction

Enter 1 if underflow can make tiny nonzero values collapse toward zero, or 2 if underflow makes them huge.

Compute it first, then check your number.