Chapter 11
Information Theory
Entropy, cross-entropy, KL divergence, mutual information, perplexity, and compression as prediction.
What this chapter does
Information theory connects probability to surprise and coding cost. This chapter explains entropy, cross-entropy, KL divergence, mutual information, perplexity, and why language-model training can be read as repeated prediction under uncertainty.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Why information theory is the mathematics of surprise.
- 02Entropy
Average uncertainty in a probability distribution.
- 03Joint and Conditional Entropy
Uncertainty together and uncertainty after context is known.
- 04Cross-Entropy
The cost of encoding targets with predicted probabilities.
- 05KL Divergence
Directional mismatch between probability distributions.
- 06Mutual Information
How much knowing one variable reduces uncertainty about another.
- 07Negative Log-Likelihood
Turning probability assigned to observed data into a loss.
- 08Perplexity
A language-model metric that behaves like average choice count.
- 09Compression as Prediction
Why better prediction usually means shorter descriptions.
- 10Classification and Language Modeling
How cross-entropy appears in classes and next-token prediction.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What Chapter 11 accomplished and how it supports later subjects.
Summary and Revision NotesA compact review of uncertainty, surprise, and distribution mismatch.
ExercisesChapter-level practice for information theory.
HintsLow-spoiler nudges for the Chapter 11 exercises.
SolutionsExplained solutions for the Chapter 11 exercises.
Before moving on
- Understand entropy, cross-entropy, KL divergence, and perplexity.
- Connect prediction, surprise, compression, and loss.
- Explain why cross-entropy is more informative than accuracy for probability predictions.
- Read classification and language-modeling objectives with less mystery.
Where this leads
- Language Modeling
- Transformers