LLM Primer

Epsilon and Safe Division

An epsilon is a small constant added for numerical safety.

0unsafe denominator+ epsilonsmall positive guard
Epsilon moves an unsafe zero or near-zero quantity away from the danger point.

For example:

xv+ϵ\frac{x}{\sqrt{v + \epsilon}}

The epsilon helps avoid division by zero or square roots of exact zero.

Epsilon Is Not Decoration

Adding epsilon changes the computation.

It should be small enough to avoid changing the intended result much, but large enough to prevent numerical failure.

This is the main tradeoff. If epsilon is too small, it may not protect the calculation. If it is too large, it can noticeably change the value being computed.

For example, compare:

10+10−8\frac{1}{0 + 10^{-8}}

and:

10+10−2\frac{1}{0 + 10^{-2}}

Both avoid division by zero, but they do not give the same scale.

ML Reading

Epsilon constants appear in normalization, optimizers, logarithms, and probability calculations.

For example, code may use log(p + epsilon) to avoid log(0).

This does not mean every formula should get an epsilon. Use it when the formula has a real numerical danger: division by zero, square root of zero in a denominator, or logarithm of zero.

Adding epsilon without knowing the danger can hide bugs or change the meaning of the computation. A guardrail is useful only when you know what it is guarding against.

MATH-C08-T08-001Exercise: Avoid log zero

If p = 0, what does log(p) try to compute?

Enter 1 for log(0), 2 for log(1).

Compute it first, then check your number.

Hint
Substitute p = 0.
Solution

It tries to compute log(0), which is not finite. Enter 1. Adding a small epsilon can guard this specific failure mode by moving the argument away from exact zero.

MATH-C08-T08-002Exercise: Epsilon changes the computation

Does adding epsilon always leave the computed value exactly unchanged?

Answer it first, then check.

Hint

The lesson says epsilon changes the computation.

Solution

No. Epsilon is a guardrail, but it still changes the expression being computed.

MATH-C08-T08-003Exercise: Choose the danger

Which operation has the clearer numerical danger: log(0) or log(0.8)?

Answer it first, then check.

Hint

The logarithm of zero is not finite.

Solution

log(0) is the dangerous operation because it is not finite. By contrast, log(0.8) has a normal finite value, so it does not need the same guard.

MATH-C08-T08-004Exercise: Too large epsilon

If epsilon is chosen too large, can it noticeably change the intended result?

Answer it first, then check.

Hint

Compare adding 10^{-8} with adding 10^{-2} near zero.

Solution

Yes. A large epsilon can dominate the quantity it is meant to protect and change the result noticeably.

MATH-C08-T08-005Exercise: Know what epsilon guards

Enter 1 if epsilon should be added for a specific numerical danger, not as a default decoration on every formula.

Compute it first, then check your number.

Hint

Ask whether the formula risks division by zero, square root of zero, or log of zero.

Solution

Enter 1. Epsilon changes the expression, so it should protect against a known numerical danger rather than being added automatically.

Before Moving On

Epsilon is a small guardrail, not a cure for every numerical issue.