LLM Primer

Cross-Entropy

Cross-entropy measures how costly it is to encode target outcomes using a model's predicted distribution.

targetpredictioncost from target probability
Cross-entropy measures how many bits or nats the prediction spends on the target.

Formula

For target distribution (p) and prediction (q):

H(p,q)=−∑ipilog⁡qiH(p, q) = -\sum_i p_i \log q_i

If the target is one-hot, cross-entropy becomes:

−log⁡qcorrect-\log q_{\text{correct}}

This is the key ML case. The target says which class or token occurred. The model says how much probability it assigned to that target. Cross-entropy turns that assigned probability into a penalty.

If the correct class receives high probability, the penalty is small. If the correct class receives low probability, the penalty is large.

In ML

Classification and language models often train with cross-entropy because the target is a correct class or token, and the model outputs probabilities.

Cross-entropy does not only ask whether the top class is correct. It also cares how much probability the model assigned. A correct answer with probability 0.9 is better, by this loss, than the same correct answer with probability 0.51.

This is the key difference from accuracy. Accuracy sees both examples as correct. Cross-entropy still rewards the model that assigned more probability to the observed target.

MATH-C11-T04-001Exercise: Half probability

If the model assigns probability 0.5 to the correct class, what is the negative log-likelihood in base 2?

Compute it first, then check your number.

Hint

(-\log_2(0.5) = 1).

Solution

Since (0.5 = 1/2), (\log_2(0.5) = -1). The negative log is 1 bit.

MATH-C11-T04-002Exercise: One-hot target

For a one-hot target, cross-entropy depends on the probability assigned to which class?

Answer it first, then check.

Hint

The formula becomes -log q_correct.

Solution

It depends on the probability assigned to the correct, or target, class.

MATH-C11-T04-003Exercise: Lower loss

Which gives lower cross-entropy for the correct class: probability 0.9 or probability 0.1?

Compute it first, then check your number.

Hint

Higher probability on the observed target gives smaller negative log.

Solution

Probability 0.9 gives lower loss because the model assigned more probability to what actually happened.

MATH-C11-T04-004Exercise: Only top class?

Does cross-entropy care only whether the top predicted class is correct?

Answer it first, then check.

Hint

It also uses the probability assigned to the target.

Solution

No. Cross-entropy cares about the probability assigned to the target, not only the rank of the top class.

MATH-C11-T04-005Exercise: Accuracy versus loss

Enter 1 if two predictions can have the same accuracy but different cross-entropy losses.

Compute it first, then check your number.

Hint

Compare correct predictions with target probabilities 0.51 and 0.9.

Solution

Enter 1. Both predictions may be correct, but the one assigning higher probability to the target has lower cross-entropy.

Before Moving On

Cross-entropy punishes a model for assigning low probability to the observed target.