Softmax Over a Vocabulary

Softmax turns output scores into a probability distribution.

If the vocabulary is:

tea, code, music

then the model must assign probabilities that sum to 1 across these options:

P(tea | context)
P(code | context)
P(music | context)

The highest score receives the highest probability, but every vocabulary item receives some probability unless masking or numerical limits intervene.

Exercise

If a vocabulary has 3 tokens, how many probabilities does softmax produce for one prediction position?