Introduction

Information theory studies uncertainty, surprise, compression, and distribution mismatch.

That sounds abstract, but modern ML uses this language constantly. Cross-entropy is a loss. KL divergence compares distributions. Perplexity evaluates language models. Compression gives a way to think about prediction.

The Main Idea

Good prediction reduces uncertainty.

If a model assigns high probability to what actually happens, it spends less information describing the outcome. If it assigns low probability to what happens, it pays a larger penalty.

This is why the chapter belongs after probability and statistics. Probability gave us distributions. Statistics asked how much evidence a number carries. Information theory asks how costly surprise is when a model must predict, encode, or compare distributions.

The Reader's Picture

Imagine a game where you must guess the next symbol.

If every symbol is equally plausible, the next symbol carries more surprise. If one symbol is almost certain, the next symbol carries less surprise. A good model is useful because it moves probability mass toward what actually happens.

That picture will reappear in classification, next-token prediction, compression, and evaluation.

What This Chapter Covers

This chapter covers entropy, joint and conditional entropy, cross-entropy, KL divergence, mutual information, negative log-likelihood, perplexity, compression as prediction, and how these ideas appear in classification and language modeling.

The goal is not to make information theory feel mystical. The goal is to make common ML losses and metrics feel readable.

Before Moving On

Read information theory as the mathematics of surprise.