Negative Log Likelihood

Likelihood measures how much probability the model assigns to the observed answer.

If the correct next token is tea and the model assigns:

P(tea | I like) = 0.25

then the likelihood for that prediction is 0.25.

Negative log likelihood, or NLL, turns that probability into a loss:

NLL = -log(P(correct token))

High probability gives low loss. Low probability gives high loss.

Why the log appears

For a sequence, probabilities multiply. Logs turn products into sums, which are easier to optimize and compare.

Exercise

If a model assigns probability 1 to the correct token, what is -log(1)?