Softmax and Stable Softmax Review

Softmax turns logits into probabilities.

logits[2, 1, 0]softmaxprobabilitiessum = 1
Softmax turns raw logits into positive probabilities that sum to one.

For logits:

z = [z_1, z_2, z_3]

softmax computes:

p_i = exp(z_i) / sum_j exp(z_j)

The outputs are positive and sum to one.

Shift does not change softmax probabilities

If you add the same constant to every logit, the softmax probabilities stay the same.

This is why stable implementations subtract the maximum logit before exponentiating:

z_stable = z - max(z)

For:

z = [1000, 999, 998]

use:

z_stable = [0, -1, -2]

This avoids huge exponentials while preserving the probabilities.

Exercise: Stable shifted logits

Shift logits [5, 3, 1] by subtracting the maximum logit. What is the first shifted value?

Compute it first, then check your number.

HintSubtract the max

The maximum logit is 5.

SolutionWork it out

The maximum is 5. The first shifted value is 5 - 5 = 0.

Exercise: Probability sum

Softmax outputs probabilities. What should their total sum be?

Compute it first, then check your number.

HintProbability vector

Softmax normalizes the positive values.

SolutionWork it out

Softmax produces a probability distribution over classes, so the probabilities sum to 1.