Review

Key Ideas

  • Floating point: finite approximation to real numbers.
  • Rounding error: difference between exact and stored result.
  • Overflow: value too large to represent.
  • Underflow: tiny value lost near zero.
  • Conditioning: sensitivity of output to input changes.
  • Stable softmax: softmax computed after subtracting the max logit.
  • Log-sum-exp: stable form of log(sum(exp(x))).
  • Epsilon: small constant used as a numerical guardrail.
  • Exploding quantity: repeated products push a value to large scale.
  • Vanishing quantity: repeated products push a value close to zero.
  • Gradient check: comparison between backpropagation and finite difference.

Formulas to Remember

Stable softmax:

ezi−m∑jezj−m\frac{e^{z_i - m}}{\sum_j e^{z_j - m}}

Log-sum-exp:

log⁡∑iexi=m+log⁡∑iexi−m\log\sum_i e^{x_i} = m + \log\sum_i e^{x_i - m}

Centered finite difference:

f(x+h)−f(x−h)2h\frac{f(x + h) - f(x - h)}{2h}

How To Read Stable Code

When implementation code differs from the formula in a textbook, ask what intermediate value became dangerous.

  • exp(large number) suggests overflow.
  • products of many tiny probabilities suggest underflow.
  • subtracting nearly equal large values suggests cancellation.
  • division by zero or log(0) suggests an epsilon guardrail.
  • suspicious gradients suggest a finite-difference check on a small example.

Stable rewrites usually preserve the mathematical quantity while changing the route taken through machine arithmetic.

Common Machine Learning Patterns

  • Subtract max before softmax: keep the largest shifted exponential at e0=1e^0 = 1.
  • Use log-sum-exp: avoid direct exponentials at unsafe scale.
  • Add epsilon inside a square-root denominator: keep the denominator away from zero.
  • Avoid long probability products: prevent underflow toward zero.
  • Clip gradients: limit extreme gradients after they appear.
  • Run gradient checks: debug a gradient implementation on a small case.

Checks Before You Move On

  • Assuming computer arithmetic is exact real arithmetic.
  • Computing softmax by exponentiating large logits directly.
  • Ignoring underflow in long products of probabilities.
  • Adding epsilon without understanding what it changes.
  • Treating ill-conditioning as always being a code bug.
  • Forgetting to add the max back in log-sum-exp.
  • Assuming stable softmax changes the model's probabilities.
  • Trusting gradients without testing a small case.
  • Comparing floating-point results with exact equality when a tolerance is more appropriate.
  • Adding epsilon by habit without identifying the numerical danger.
  • Treating gradient clipping as a full fix rather than a guardrail.

Mental Model

Stable numerical code preserves the mathematical idea while avoiding avoidable machine failures.