Joint and Conditional Entropy
Joint entropy measures uncertainty about two variables together.
Conditional entropy measures uncertainty about one variable after another variable is known.
The Relationship
A useful identity is:
Read it as:
uncertainty about both equals uncertainty about X, plus remaining uncertainty about Y after X is known.
If X tells us a lot about Y, then H(Y | X) is small. If X tells us
almost nothing about Y, then H(Y | X) is close to H(Y).
The order matters. H(Y | X) asks about uncertainty in Y after knowing X.
It is not the same phrase as H(X | Y), even when the two happen to be equal in
some special cases.
In ML
Conditional entropy helps express prediction. A model tries to reduce uncertainty about the target after seeing the input.
For language modeling, the model estimates uncertainty about the next token given previous tokens.
For classification, X can be the input and Y the label. A useful input is
one that leaves less uncertainty about the label.
Conditional entropy is therefore a way to name the value of context. Good context does not merely add information; it removes uncertainty about the thing we are trying to predict.
If H(X) = 2 and H(Y | X) = 3, what is H(X, Y)?
Compute it first, then check your number.
Hint
Use the identity above.
Solution
The joint uncertainty is the uncertainty in X plus the uncertainty still left
in Y after X is known. Here those two parts add to 5 bits.
If knowing X makes Y easier to predict, should H(Y | X) be smaller than
H(Y)?
Answer it first, then check.
Hint
Conditional entropy is remaining uncertainty after context is known.
Solution
Yes. If X gives useful context about Y, then after observing X there is
less uncertainty left about Y. That means H(Y | X) should be lower than the
original H(Y).
If X tells us nothing about Y, then H(Y | X) is close to which quantity?
Answer it first, then check.
Hint
If the context gives no help, uncertainty does not drop.
Solution
It stays close to H(Y), the original uncertainty about Y. If X tells us
nothing useful, conditioning on X does not remove much uncertainty.
Does H(Y | X) read as uncertainty in Y after knowing X?
Answer it first, then check.
Hint
Read the expression from right context to left target.
Solution
Yes. In H(Y | X), the left side of the bar is the target whose uncertainty we
measure, and the right side is the context we already know. It means uncertainty
in Y after X is known.
Enter 1 if useful context lowers the remaining uncertainty about the target.
Compute it first, then check your number.
Hint
Ask what should happen to H(Y | X) when X helps predict Y.
Solution
Enter 1. If X helps predict Y, then less uncertainty remains about Y
after X is known.
Before Moving On
Conditional entropy is what remains uncertain after using available context.