LLM Primer

Log-Sum-Exp

Log-sum-exp appears when computing log probabilities and normalizers.

The direct form is:

log(exp(a) + exp(b) + ...)

The stable form subtracts the maximum before exponentiating:

m = max(values)
log_sum_exp(values) = m + log(sum(exp(values - m)))
values3, 9, 4subtract max-6, 0, -5exp and sumsafe scaleadd max backsame resultlargest shifted value is 0
Log-sum-exp shifts first, exponentiates safely, then restores the scale.

Subtracting m does not change the mathematical result. It changes the intermediate numbers so they are less likely to overflow.

For values [1000, 1001], direct exponentials are dangerous. But subtracting 1001 gives:

[-1, 0]

Those exponentials are safe.

DL-C16-T04-001Exercise: Max shift

For values [5, 9, 7], what maximum m should be subtracted in the stable log-sum-exp rewrite?

Compute it first, then check your number.

DL-C16-T04-002Exercise: Shifted value

Values are [5, 9] and m = 9. What is the first shifted value?

Compute it first, then check your number.