Summary and Revision Notes

Key Ideas

Idea	Meaning
entropy	average uncertainty in a distribution
conditional entropy	uncertainty remaining after context is known
cross-entropy	cost of encoding targets using predicted probabilities
KL divergence	directional mismatch between distributions
mutual information	uncertainty reduction shared between variables
negative log-likelihood	loss from probability assigned to observed data
perplexity	effective average choice count for prediction
compression	fewer bits when prediction is better
next-token prediction	classification over possible next tokens

The recurring pattern is probability assigned to what happened. Information theory gives different names to different ways of reading that assignment: uncertainty before the outcome, cost after the outcome, mismatch between distributions, or effective choice count.

Formulas to Remember

Entropy:

H(p) = -\sum_i p_i \log p_i

Cross-entropy:

H(p, q) = -\sum_i p_i \log q_i

KL divergence:

D_{\mathrm{KL}}(p \parallel q) = \sum_i p_i \log \frac{p_i}{q_i}

Mutual information:

I(X;Y) = H(Y) - H(Y \mid X)

Joint entropy identity:

H(X, Y) = H(X) + H(Y \mid X)

Negative log-likelihood:

\mathrm{NLL} = -\log p_{\text{observed}}

Perplexity from bits:

\mathrm{perplexity} = 2^{\text{loss in bits}}

Common Traps

Treating KL divergence as symmetric.
Forgetting the log base changes the unit.
Confusing low entropy with low loss in every context.
Assuming low entropy means the prediction is correct.
Forgetting that cross-entropy uses the probability assigned to the target, not just whether the top class is correct.
Treating mutual information as proof of causation.
Treating perplexity as meaningful without knowing the dataset and tokenizer.
Saying compression explains everything about intelligence.
Reading entropy as accuracy.
Forgetting that cross-entropy can distinguish two correct predictions.
Forgetting that perplexity is an exponentiated average loss.
Treating next-token prediction as trivial because one step is simple.

Reading ML Reports

When reading a loss or metric, ask:

What distribution is being measured?
Which outcome was observed?
What probability did the model assign to that outcome?
Is the number in bits, nats, or a derived scale like perplexity?
Are the dataset and tokenizer comparable to the system being discussed?
Is the claim about prediction quality, distribution mismatch, or broader model behavior?

Mental Model

Information theory asks how expensive surprise is.