LLM Primer

Summary and Revision Notes

Key Ideas

IdeaMeaning
floating pointfinite approximation to real numbers
rounding errordifference between exact and stored result
overflowvalue too large to represent
underflowtiny value lost near zero
conditioningsensitivity of output to input changes
stable softmaxsoftmax computed after subtracting max logit
log-sum-expstable form of log(sum(exp(x)))
epsilonsmall constant used as a numerical guardrail
exploding quantityrepeated products push a value to large scale
vanishing quantityrepeated products push a value close to zero
gradient checkcomparison between backprop and finite difference

Formulas to Remember

Stable softmax:

ezi−m∑jezj−m\frac{e^{z_i - m}}{\sum_j e^{z_j - m}}

Log-sum-exp:

log⁡∑iexi=m+log⁡∑iexi−m\log\sum_i e^{x_i} = m + \log\sum_i e^{x_i - m}

Centered finite difference:

f(x+h)−f(x−h)2h\frac{f(x + h) - f(x - h)}{2h}

How To Read Stable Code

When implementation code differs from the formula in a textbook, ask what intermediate value became dangerous.

  • exp(large number) suggests overflow.
  • products of many tiny probabilities suggest underflow.
  • subtracting nearly equal large values suggests cancellation.
  • division by zero or log(0) suggests an epsilon guardrail.
  • suspicious gradients suggest a finite-difference check on a small example.

Stable rewrites usually preserve the mathematical quantity while changing the route taken through machine arithmetic.

Common ML Patterns

PatternNumerical reason
subtract max before softmaxkeep largest shifted exponential at e^0 = 1
use log-sum-expavoid direct exponentials at unsafe scale
add epsilon in denominatorsavoid division by zero or square root of zero
avoid long probability productsprevent underflow toward zero
clip gradientslimit extreme gradients after they appear
run gradient checksdebug a gradient implementation on a small case

Common Traps

  • Assuming computer arithmetic is exact real arithmetic.
  • Computing softmax by exponentiating large logits directly.
  • Ignoring underflow in long products of probabilities.
  • Adding epsilon without understanding what it changes.
  • Treating ill-conditioning as always being a code bug.
  • Forgetting to add the max back in log-sum-exp.
  • Assuming stable softmax changes the model's probabilities.
  • Trusting gradients without testing a small case.
  • Comparing floating-point results with exact equality when a tolerance is more appropriate.
  • Adding epsilon by habit without identifying the numerical danger.
  • Treating gradient clipping as a full fix rather than a guardrail.

Mental Model

Stable numerical code preserves the mathematical idea while avoiding avoidable machine failures.