Entropy
Entropy measures average uncertainty.
When one outcome is almost certain, entropy is low. When many outcomes are plausible, entropy is higher.
Formula
For probabilities (p_i), entropy is:
Here p_i is the probability of outcome i. The expression log p_i is
negative when 0 < p_i < 1, so the minus sign makes the contribution positive.
When p_i = 0, the term is treated as contributing 0.
The log base controls the unit. Base 2 gives bits. Natural log gives nats.
Small Example
A fair coin has two equally likely outcomes. In bits, its entropy is 1 bit.
Why 1 bit? One yes-or-no question can identify the outcome: heads or tails.
A certain outcome has entropy 0 because there is nothing to identify. An impossible alternative does not add uncertainty if the result is already known.
In ML
Entropy describes uncertainty in distributions. A classifier output near
[0.99, 0.01] has low uncertainty. A classifier output near [0.5, 0.5] has
higher uncertainty.
High entropy is not automatically bad. It may honestly reflect ambiguity in the input. Low entropy is not automatically good. A model can be confidently wrong.
Entropy is about the distribution before the outcome is revealed. It does not ask whether the model's favorite answer matches the truth. Accuracy and loss ask different questions.
If a certain event has probability 1, what is its uncertainty in bits?
Compute it first, then check your number.
Hint
If there is no uncertainty, no information is needed to identify the outcome.
Solution
A certain event has no surprise because the outcome is already known before it happens. In information terms, no yes-or-no questions are needed, so the uncertainty is 0 bits.
How many bits of entropy does a fair coin have?
Compute it first, then check your number.
Hint
One yes-or-no question identifies heads or tails.
Solution
A fair coin has two equally likely outcomes. One binary answer, heads or tails, is enough to identify the outcome, so its entropy is 1 bit.
Which distribution has higher uncertainty: [0.99, 0.01] or [0.5, 0.5]?
Answer it first, then check.
Hint
Entropy is higher when plausible outcomes are more evenly spread.
Solution
[0.5, 0.5] has higher uncertainty because neither outcome dominates. In
[0.99, 0.01], the next outcome is almost known in advance, so there is much
less uncertainty left.
Does low entropy always mean a model is correct?
Answer it first, then check.
Hint
A model can be confident and still assign high probability to the wrong class.
Solution
No. Low entropy means the model is confident. It does not by itself prove the prediction is correct.
Enter 1 if entropy measures uncertainty in a distribution, not whether the
model's chosen class is correct.
Compute it first, then check your number.
Hint
Think about a model that is very confident in the wrong answer.
Solution
Enter 1. Entropy measures how spread out the probability distribution is. It
does not by itself compare the prediction with the true label.
Before Moving On
Entropy is not disorder in a vague sense. Here it is average uncertainty in a probability distribution.