Hints

MATH-C11-C-001

A certain event has no uncertainty.

Use H(X, Y) = H(X) + H(Y | X).

0.125 = 1/8 = 2^-3.

Use 2 raised to the loss measured in bits.

Use I(X;Y) = H(Y) - H(Y | X).

For one-hot targets, the formula becomes -log q_correct.

Symmetric would mean swapping the two distributions leaves the value unchanged.

The unit of prediction depends on tokenization.

Shared information and causal influence are different claims.

At each position, the model predicts a distribution over possible next tokens.

Confidence and correctness are different questions.

Accuracy sees whether the top answer is right. Cross-entropy reads the target probability.

Look at which distribution weights the expectation.

The prediction unit depends on the tokenizer.

Predicting text well is not the same as satisfying every downstream requirement.