LLM Primer

Mutual Information

Mutual information measures how much knowing one variable reduces uncertainty about another.

XYshared information
Mutual information asks how much knowing one variable reduces uncertainty about another.

Formula

One way to write it is:

I(X;Y)=H(Y)−H(Y∣X)I(X;Y) = H(Y) - H(Y \mid X)

If knowing (X) reduces uncertainty about (Y), mutual information is larger.

If X tells us nothing useful about Y, then H(Y | X) is about the same as H(Y), so mutual information is near 0.

If X tells us a lot about Y, then H(Y | X) is much smaller, so mutual information is larger.

In ML

Mutual information helps describe representation learning, bottlenecks, dependence between variables, and how much a feature tells us about a target.

It is about information shared between variables. It is not the same as saying one variable causes the other.

Mutual information also ignores the sign of a relationship. It tells us that knowing one variable changes uncertainty about another, not whether the relationship is positive, negative, linear, or causal.

MATH-C11-T06-001Exercise: Information reduction

If H(Y) = 5 and H(Y | X) = 2, what is I(X;Y)?

Compute it first, then check your number.

Hint

Subtract remaining uncertainty from original uncertainty.

Solution
I(X;Y)=H(Y)−H(Y∣X)=5−2=3I(X;Y) = H(Y) - H(Y \mid X) = 5 - 2 = 3

Knowing X removes 3 bits of uncertainty about Y. That removed uncertainty is the mutual information.

MATH-C11-T06-002Exercise: No reduction

If H(Y) = 5 and H(Y | X) = 5, what is I(X;Y)?

Compute it first, then check your number.

Hint

Subtract remaining uncertainty from original uncertainty.

Solution

I(X;Y) = 5 - 5 = 0. Knowing X did not reduce uncertainty about Y, so the two variables share no useful information under this measurement.

MATH-C11-T06-003Exercise: Shared information

Does larger mutual information mean X reduces more uncertainty about Y?

Answer it first, then check.

Hint

Mutual information measures uncertainty reduction.

Solution

Yes. Larger mutual information means knowing one variable reduces more uncertainty about the other.

MATH-C11-T06-004Exercise: Causation?

Does mutual information by itself prove that X causes Y?

Answer it first, then check.

Hint

Shared information and causation are different claims.

Solution

No. Mutual information describes dependence or shared information. It does not by itself prove causation.

MATH-C11-T06-005Exercise: Dependence, not direction

Enter 1 if mutual information can show dependence without telling us the causal direction.

Compute it first, then check your number.

Hint

Knowing two variables move together is not the same as knowing which one caused the other.

Solution

Enter 1. Mutual information measures shared information or dependence. It does not identify causal direction by itself.

Before Moving On

Mutual information is about shared information, not simple correlation alone.