Review

Key Ideas

  • Entropy: average uncertainty in a distribution.
  • Conditional entropy: uncertainty remaining after context is known.
  • Cross-entropy: cost of encoding targets using predicted probabilities.
  • KL divergence: directional mismatch between distributions.
  • Mutual information: uncertainty reduction shared between variables.
  • Negative log-likelihood: loss from probability assigned to observed data.
  • Perplexity: effective average choice count for prediction.
  • Compression: fewer bits when prediction is better.
  • Next-token prediction: classification over possible next tokens.

The recurring pattern is probability assigned to what happened. Information theory gives different names to different ways of reading that assignment: uncertainty before the outcome, cost after the outcome, mismatch between distributions, or effective choice count.

Formulas to Remember

Entropy:

H(p)=−∑ipilog⁡piH(p) = -\sum_i p_i \log p_i

Cross-entropy:

H(p,q)=−∑ipilog⁡qiH(p, q) = -\sum_i p_i \log q_i

KL divergence:

DKL(p∥q)=∑ipilog⁡piqiD_{\mathrm{KL}}(p \parallel q) = \sum_i p_i \log \frac{p_i}{q_i}

Mutual information:

I(X;Y)=H(Y)−H(Y∣X)I(X;Y) = H(Y) - H(Y \mid X)

Joint entropy identity:

H(X,Y)=H(X)+H(Y∣X)H(X, Y) = H(X) + H(Y \mid X)

Negative log-likelihood:

NLL=−log⁡pobserved\mathrm{NLL} = -\log p_{\text{observed}}

Perplexity from bits:

perplexity=2average per-token loss in bits\mathrm{perplexity} = 2^{\text{average per-token loss in bits}}

Checks Before You Move On

  • Treating KL divergence as symmetric.
  • Forgetting the log base changes the unit.
  • Confusing low entropy with low loss in every context.
  • Assuming low entropy means the prediction is correct.
  • Forgetting that cross-entropy uses the probability assigned to the target, not just whether the top class is correct.
  • Treating mutual information as proof of causation.
  • Treating perplexity as meaningful without knowing the dataset and tokenizer.
  • Saying compression explains everything about intelligence.
  • Reading entropy as accuracy.
  • Forgetting that cross-entropy can distinguish two correct predictions.
  • Forgetting that perplexity is an exponentiated average loss.
  • Treating next-token prediction as trivial because one step is simple.
  • Forgetting that cross-entropy equals target entropy plus directional KL divergence.
  • Assigning zero model probability where the reference distribution has positive probability, which makes forward KL infinite.
  • Exponentiating total sequence NLL instead of average per-token NLL when computing perplexity.

Reading Machine Learning Reports

When reading a loss or metric, ask:

  • What distribution is being measured?
  • Which outcome was observed?
  • What probability did the model assign to that outcome?
  • Is the number in bits, nats, or a derived scale like perplexity?
  • Are the dataset and tokenizer comparable to the system being discussed?
  • Is the claim about prediction quality, distribution mismatch, or broader model behavior?

Mental Model

Information theory asks how expensive surprise is.