Exercises
These exercises review the chapter as a whole. They mix small computations with the interpretation mistakes that often make ML losses and metrics confusing.
In bits, a fair coin has entropy 1.
How many bits of entropy does a certain event have?
Compute it first, then check your number.
If H(X) = 4 and H(Y | X) = 2, what is H(X, Y)?
Compute it first, then check your number.
In base 2, what is -log2(0.125)?
Compute it first, then check your number.
If average loss is 4 bits, what is perplexity?
Compute it first, then check your number.
If H(Y) = 7 and H(Y | X) = 3, what is I(X;Y)?
Compute it first, then check your number.
For a one-hot target, cross-entropy reads the probability assigned to which class?
Answer it first, then check.
Is KL divergence generally symmetric?
Answer it first, then check.
Should perplexity values from different tokenizers be compared casually?
Answer it first, then check.
Does mutual information by itself prove causation?
Answer it first, then check.
Can next-token prediction be read as classification over a vocabulary?
Answer it first, then check.
Enter 1 if a low-entropy prediction can still be wrong.
Compute it first, then check your number.
Enter 1 if two correct predictions can have the same accuracy but different
cross-entropy losses.
Compute it first, then check your number.
Enter 1 if D_KL(p || q) measures mismatch from p's weighting view.
Compute it first, then check your number.
Enter 1 if perplexity comparisons require compatible evaluation data and
tokenization.
Compute it first, then check your number.
Enter 1 if good compression or prediction is not by itself a full test of
truthfulness, safety, or usefulness.
Compute it first, then check your number.
Next
Use the hints only after you have tried the exercises.