Perplexity
Perplexity is a common language-model metric.
It behaves like the effective number of choices the model is confused among.
That phrase is an interpretation, not a new measurement. Perplexity is computed from average negative log-likelihood. It is useful because a log-loss number can feel abstract, while an effective choice count is easier to picture.
Formula
If the average negative log-likelihood is measured in natural logs, perplexity is:
If the average loss is measured in bits, perplexity is:
Small Example
If average loss is 2 bits, perplexity is:
This can be read as: the model behaves as if it is choosing among about 4 equally likely options on average.
Perplexity depends on the evaluation data and tokenization. Do not compare perplexity numbers across incompatible tokenizers or datasets as if they were the same measurement.
Perplexity is an exponentiated average loss. That makes it easier to read as an effective choice count, but it also hides the log scale underneath. Always ask what loss, tokenizer, and dataset produced it.
If average loss is 3 bits, what is perplexity?
Compute it first, then check your number.
Hint
Use 2 raised to the loss measured in bits.
Solution
With loss measured in bits, perplexity is 2^loss. A 3-bit average loss
therefore corresponds to an effective choice count of 8.
If average loss is 2 bits, what is perplexity?
Compute it first, then check your number.
Hint
Use 2^loss when loss is measured in bits.
Solution
2^2 = 4, so perplexity is 4. This means the model behaves, on average, as if
it is choosing among about four equally likely options.
On the same dataset with the same tokenizer, does lower perplexity usually mean better next-token prediction?
Answer it first, then check.
Hint
Lower perplexity comes from lower average negative log-likelihood.
Solution
Yes. On the same evaluation setup, lower perplexity usually means better next-token prediction.
Should perplexity numbers from different tokenizers be compared casually?
Answer it first, then check.
Hint
The unit of prediction depends on the tokenization.
Solution
No. Perplexity depends on tokenization and evaluation data, so incompatible setups should not be compared casually.
Enter 1 if perplexity is derived from average negative log-likelihood rather
than being a separate kind of model behavior.
Compute it first, then check your number.
Hint
Look at the formula perplexity = e^NLL or 2^loss.
Solution
Enter 1. Perplexity is computed from average negative log-likelihood. It is a
readable scale for the same prediction-cost idea.
Model A has lower perplexity than Model B on the same validation set with the same tokenizer. Is that meaningful evidence that Model A assigned better probabilities on that setup?
Answer it first, then check.
Hint
The comparison is controlled: same data, same tokenizer.
Solution
Yes. On the same data with the same tokenizer, lower perplexity means lower average negative log-likelihood, so Model A assigned better probabilities under that evaluation setup.
Before Moving On
Lower perplexity usually means better next-token prediction on the evaluated data.