Summary and Revision Notes

Core ideas

• Negative log likelihood is -log(P(correct token)).
• Cross-entropy is average negative log likelihood.
• Perplexity is exponentiated cross-entropy.
• Bits per token depends on the tokenizer.
• Bits per byte can make some comparisons less tokenizer-dependent.
• Memorization and leakage can make evaluation too optimistic.

Check yourself

• Can you compute an average loss?
• Can you convert 3 bits of cross-entropy into perplexity?
• Can you explain why a score without dataset and tokenizer details is weak?