LLM Primer

Summary and Revision Notes

Key ideas

  • Floating-point computation is approximate.
  • Dtype affects range, precision, memory, and speed.
  • FP32 is a common stable default.
  • FP16 saves memory but has narrower range.
  • BF16 keeps a wider range with less precision.
  • Overflow makes values too large to represent.
  • Underflow can collapse tiny nonzero values toward zero.
  • Stable softmax subtracts the maximum logit.
  • Log-sum-exp subtracts the maximum before exponentiating.
  • Epsilons guard small denominators.
  • NaN and infinity should trigger debugging.
  • Mixed precision uses dtype choices deliberately.

Common formulas

stable_softmax(logits) =
    softmax(logits - max(logits))
log_sum_exp(x) =
    m + log(sum(exp(x - m)))
    where m = max(x)
safe_denominator = denominator + epsilon

Common mistakes

  • Treating numerical stability as separate from model behavior.
  • Computing softmax directly on very large logits.
  • Adding epsilon without understanding the denominator.
  • Ignoring the first NaN and continuing training.
  • Assuming lower precision is always safe because it is faster.

Before moving on

You should be able to identify overflow, underflow, unsafe softmax, unsafe division, NaN propagation, and the basic reason mixed precision needs care.