KL Divergence

KL divergence measures how much one distribution differs from another.

KL divergence is directional: it measures mismatch from a reference distribution's view.

Formula

For distributions (p) and (q):

D_{\mathrm{KL}}(p \parallel q) = \sum_i p_i \log \frac{p_i}{q_i}

Read this as: measure how expensive it is to use q when the reference distribution is p.

This is directional. In general:

D_{\mathrm{KL}}(p \parallel q) \ne D_{\mathrm{KL}}(q \parallel p)

The direction matters because the expectation is weighted by p. Mistakes on outcomes that p considers likely count more in D_KL(p || q).

In ML

KL divergence appears in variational methods, distillation, RL, language-model alignment, and regularization between distributions.

It is not a distance in the usual geometric sense. Distances are symmetric and usually satisfy a triangle inequality. KL divergence is better read as a directional mismatch.

Another useful reading is extra coding cost. If p describes what really happens and q is the code or model we use instead, D_KL(p || q) measures the extra expected cost from using q under p's view.

MATH-C11-T05-001Exercise: Symmetry

Enter 1 if KL divergence is generally symmetric. Enter 0 otherwise.

Check

Compute it first, then check your number.

Hint

Symmetric would mean swapping (p) and (q) gives the same value.

Solution

Enter 0. KL divergence is generally not symmetric because swapping p and q changes which distribution supplies the weighting. The two directions can therefore assign different costs to the same mismatch.

MATH-C11-T05-002Exercise: Reference distribution

In D_KL(p || q), which distribution supplies the weighting view: p or q?

Check

Answer it first, then check.

Hint

Look at the sum: each term is weighted by p_i.

Solution

p supplies the weighting view in D_KL(p || q). Each term is weighted by p_i, so outcomes that are likely under p matter more when measuring how well q matches that reference distribution.

MATH-C11-T05-003Exercise: Distance?

Is KL divergence a distance in the usual symmetric geometric sense?

Check yes or no

Answer it first, then check.

Hint

The lesson says KL is directional.

Solution

No. KL divergence is directional, so it is not a usual symmetric distance.

MATH-C11-T05-004Exercise: Same distribution

If p and q are exactly the same distribution, what is D_KL(p || q)?

Check

Compute it first, then check your number.

Hint

If there is no mismatch, the extra cost is zero.

Solution

The KL divergence is 0 when the two distributions match exactly. In that case each probability ratio is 1, log(1) = 0, and there is no extra mismatch cost.

MATH-C11-T05-005Exercise: Extra cost view

Enter 1 if D_KL(p || q) can be read as an extra expected cost from using q when p is the reference distribution.

Your answer

Compute it first, then check your number.

Hint

Look at which distribution weights the expectation in D_KL(p || q).

Solution

Enter 1. In D_KL(p || q), outcomes are weighted by p, so the mismatch is measured from p's reference view.

Before Moving On

KL divergence is a distribution mismatch measured from a chosen reference view.