LLM Primer

Log-Sum-Exp

Log-sum-exp is the stable way to compute:

log⁡∑iexi\log\sum_i e^{x_i}

The direct computation can overflow because of e^{x_i}.

values3, 9, 4subtract max-6, 0, -5exp and sumsafe scaleadd max backsame resultlargest shifted value is 0
Log-sum-exp shifts first, exponentiates safely, then restores the scale.

Stable Form

Let:

m=max⁡ixim = \max_i x_i

Then:

log⁡∑iexi=m+log⁡∑iexi−m\log\sum_i e^{x_i} = m + \log\sum_i e^{x_i - m}

The shifted exponentials are safer because the largest shifted value is 0.

The added m restores the scale that was removed before exponentiation. Without adding it back, the result would describe the shifted numbers, not the original numbers.

A Small Example

For [3, 9, 4], the maximum is 9.

The shifted values are [-6, 0, -5].

Then:

log⁡(e3+e9+e4)=9+log⁡(e−6+e0+e−5)\log(e^3 + e^9 + e^4) = 9 + \log(e^{-6} + e^0 + e^{-5})

The largest exponential on the right is e^0 = 1, which is safe.

ML Reading

Log-sum-exp appears in softmax, cross-entropy, likelihoods, and probabilistic models.

It is a core example of stable numerical rewriting.

The stable form is useful for the same reason stable softmax is useful: it controls exponentials before they can overflow. The added maximum is what keeps the final answer tied to the original scale.

MATH-C08-T07-001Exercise: Find the shift

For values [3, 9, 4], what is m in the stable log-sum-exp formula?

Compute it first, then check your number.

Hint
m is the maximum value.
Solution

The maximum is 9. Choosing the maximum as the shift makes every shifted value less than or equal to zero, which keeps the exponentials safer.

MATH-C08-T07-002Exercise: Largest shifted value

For values [3, 9, 4], what is the largest value after subtracting m = 9?

Compute it first, then check your number.

Hint

The maximum value is subtracted from itself.

Solution

The largest shifted value is 9 - 9 = 0. This is the key safety property: after shifting, the largest exponential is e^0 = 1.

MATH-C08-T07-003Exercise: Restore the scale

In the stable log-sum-exp formula, do we add m back after taking the log?

Answer it first, then check.

Hint

The shift changed the scale before exponentiation.

Solution

Yes. We add m back so the result matches the original unshifted expression.

MATH-C08-T07-004Exercise: Why the rewrite helps

Why is the shifted form safer?

Enter 1 for because the largest shifted exponential is e^0, or 2 for because logarithms stop needing exponentials.

Compute it first, then check your number.

Hint

The formula still uses exponentials, but on shifted values.

Solution

The largest shifted value is 0, so the largest shifted exponential is e^0 = 1. Enter 1.

MATH-C08-T07-005Exercise: Do not forget the max

Enter 1 if forgetting to add m back would compute the log-sum-exp of the shifted values, not the original values.

Compute it first, then check your number.

Hint

Ask what quantity remains if the removed scale is never restored.

Solution

Enter 1. Subtracting m makes exponentiation safer, but adding m back is needed to recover the original unshifted log-sum-exp.

Before Moving On

Log-sum-exp is stable because it shifts before exponentiating.