LLM Primer

Conclusion

This chapter introduced numerical computation as the bridge between exact formulas and machine arithmetic.

The point is not to distrust computers. The point is to remember that code computes with finite formats, finite ranges, and rounded intermediate values.

You learned that:

  • floating point stores finite approximations
  • rounding error can accumulate and cancellation can remove useful digits
  • overflow and underflow are scale failures at opposite ends of the range
  • conditioning measures how much input error can affect output error
  • stable softmax subtracts the maximum logit without changing probabilities
  • log-sum-exp shifts before exponentiating, then restores the removed scale
  • epsilon constants are guardrails that also change the computation
  • repeated products can make quantities explode or vanish
  • gradient checks compare backpropagation with finite differences on small cases

What Changed In Your Reading

Before this chapter, formulas could look like the whole story.

After this chapter, every formula has a second question attached:

Can this be computed safely for the values the model may produce?

That question explains many small-looking implementation choices in ML code: subtracting a maximum, adding epsilon, clipping gradients, checking a derivative, or avoiding a direct product of many tiny probabilities.

What Comes Next

Optimization comes next.

Numerical computation prepares it by explaining why learning rates, gradients, scales, and stable formulas matter in real training loops.

Keep This Question Nearby

When code differs from the formula, ask:

What numerical failure is this implementation avoiding?

Sometimes the answer is overflow. Sometimes it is underflow, cancellation, division by zero, bad scaling, or an expensive check that should only be used for debugging.

The second habit is to ask what tradeoff the fix introduces. A stable rewrite may preserve the exact mathematical quantity. An epsilon, a tolerance, or gradient clipping changes a decision boundary in the computation. These tools are useful because they are deliberate, not because they are invisible.