Cross-Entropy

Cross-entropy is the average negative log likelihood over examples.

For language modeling, each predicted token contributes a loss. The average tells us how surprised the model is, on average, by the true next token.

If four token predictions have NLL values:

1, 2, 1, 4

then the cross-entropy is:

(1 + 2 + 1 + 4) / 4 = 2

The units depend on the log base. Natural logs give nats. Base-2 logs give bits.

Exercise

What is the average of 1, 2, 1, 4?