Negative Log-Likelihood
Negative log-likelihood turns probability assigned to observed data into a loss.
If the model assigns probability (p) to the correct outcome, the loss is:
This loss is large when p is small and small when p is large. That is the
behavior we want when training a probabilistic model: reward probability placed
on what happened, punish probability withheld from what happened.
Why Negative?
Likelihood is something we maximize.
Loss is something we minimize.
Taking the negative log turns high probability into low loss and low probability into high loss.
The log also turns products into sums. For a sequence of observed tokens, the likelihood multiplies token probabilities. The negative log-likelihood adds one penalty per token.
That additive form matters because every observed token contributes a cost. Training does not only notice whether a whole sequence was likely. It can assign credit and blame across the token-level probabilities that made the sequence.
Small Example
In base 2:
If the model assigns probability 1, the loss is 0. If it assigns probability
near 0, the loss becomes very large.
In base 2, what is -log2(0.25)?
Compute it first, then check your number.
Hint
(0.25 = 1/4 = 2^-2).
Solution
(0.25 = 2^-2), so (\log_2(0.25) = -2). Negative log-likelihood takes the negative of that value, giving a loss of 2 bits.
If the model assigns probability 1 to the observed outcome, what is the
negative log-likelihood?
Compute it first, then check your number.
Hint
The log of 1 is 0.
Solution
The negative log-likelihood is 0 because log(1) = 0, so -log(1) = 0. A
model that assigns probability 1 to the observed outcome has no surprise cost
for that observation.
Which probability gives a larger negative log-likelihood: 0.1 or 0.8?
Compute it first, then check your number.
Hint
Lower assigned probability to the observed outcome gives higher loss.
Solution
Probability 0.1 gives the larger loss because the model assigned little
probability to what happened.
Do logs turn products of probabilities into sums of log-probabilities?
Answer it first, then check.
Hint
This is one reason negative log-likelihood is convenient for sequences.
Solution
Yes. Logs turn products into sums, so sequence likelihood can become a sum of token-level losses.
Enter 1 if sequence negative log-likelihood can be read as a sum of
token-level surprise costs.
Compute it first, then check your number.
Hint
Think of each observed token contributing one -log p term.
Solution
Enter 1. For a sequence, multiplying token probabilities becomes adding
log-probabilities, so NLL can be read as a sum of token-level costs.
Before Moving On
Negative log-likelihood is the bridge between probability and optimization.