Perplexity

Perplexity converts average loss into a rough branching-factor intuition.

With natural logs:

perplexity = exp(cross-entropy)

With base-2 logs:

perplexity = 2^(cross-entropy in bits)

If cross-entropy is 2 bits per token, then:

perplexity = 2^2 = 4

This can be read loosely as: the model is as uncertain as choosing among about 4 equally likely options at each step.

Perplexity is useful, but it is not the whole story. Tokenizers differ, datasets differ, and lower perplexity does not guarantee better reasoning or usefulness.

Exercise

If cross-entropy is 3 bits per token, what is the perplexity?