LLM Primer

Chapter 8

Numerical Computation

Floating point, rounding, overflow, underflow, stable softmax, log-sum-exp, and gradient checks.

What this chapter does

Numerical computation explains what changes when exact formulas run on finite machines. This chapter shows why stable code can look different from the algebra while preserving the same mathematical intent.

Lessons

Read these in order.

Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.

  1. 01
    Introduction

    Why exact formulas need care on finite machines.

  2. 02
    Floating-Point Numbers

    Finite approximations to real numbers.

  3. 03
    Rounding Error

    Small stored errors and how they can accumulate.

  4. 04
    Overflow and Underflow

    Scale failures at the large and tiny ends of numeric range.

  5. 05
    Conditioning

    How sensitive a problem is to small input changes.

  6. 06
    Stable Softmax

    Computing softmax safely by subtracting the maximum logit.

  7. 07
    Log-Sum-Exp

    A stable form for log of summed exponentials.

  8. 08
    Epsilon and Safe Division

    Small constants used as numerical guardrails.

  9. 09
    Exploding and Vanishing Quantities

    Repeated products that grow or shrink quickly.

  10. 10
    Gradient Checks

    Testing gradients with finite-difference estimates.

Review and practice

Close the chapter deliberately.

Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.

Conclusion

What Chapter 8 accomplished and how it prepares optimization.

Summary and Revision Notes

A compact review of numerical failures and stable rewrites.

Exercises

Chapter-level practice for stable numerical computation.

Hints

Low-spoiler nudges for the Chapter 8 exercises.

Solutions

Explained solutions for the Chapter 8 exercises.

Before moving on

  • Recognize common numerical failures.
  • Understand why stable implementations differ from naive formulas.
  • Spot scale, cancellation, and conditioning problems before blaming the model.
  • Use gradient checks as small tests for trust.

Where this leads

  • Optimization
  • Deep Learning

Chapter progress