Introduction

Numerical computation asks what happens when exact formulas are computed by a finite machine.

Mathematics often writes real numbers as if they have unlimited precision. Computers do not store unlimited precision. They store finite approximations.

Why This Matters

In ML, small numerical choices can decide whether training works.

• exponentials can overflow
• tiny probabilities can underflow
• subtraction can lose useful digits
• gradients can vanish or explode
• checks can pass in theory and fail in code

The problem is not that computers are bad at arithmetic. The problem is that real-number mathematics assumes unlimited precision, while machines use finite formats and fixed ranges.

That difference matters most when computations are repeated many times or when numbers become extremely large or extremely small.

What This Chapter Covers

This chapter introduces floating point, rounding error, overflow, underflow, conditioning, stable softmax, log-sum-exp, epsilon constants, exploding and vanishing quantities, and gradient checks.

The Main Question

When you see a formula in code, ask:

Will this computation stay accurate and finite for the values the model may produce?

That question is why we shift logits before softmax, use log-sum-exp, add small epsilon values carefully, and check gradients before trusting a training loop.

Before Moving On

You should be comfortable with vectors, matrices, exponentials, logarithms, probability distributions, and gradients.