Chapter 8

Evaluating Language Models

Negative log likelihood, cross-entropy, perplexity, bits per token, validation text, memorization, leakage, and evaluation limits.

What this chapter does

Evaluation connects prediction quality to measurable quantities and exposes where measurements can mislead.

Lessons

Read these in order.

Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.

01
Introduction
Evaluation as probability assigned to held-out text.
02
Negative Log Likelihood
Turning probability assigned to the correct token into a loss.
03
Cross-Entropy
Average negative log likelihood across token predictions.
04
Perplexity
Exponentiated cross-entropy as a rough branching-factor intuition.
05
Bits per Token and Byte
Why units and tokenizers matter when comparing language-model scores.
06
Memorization, Leakage, and Limits
What evaluation can measure, and what one score cannot prove.

Review and practice

Close the chapter deliberately.

Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.

Conclusion

What evaluation establishes before embeddings and neural language models.

Summary and Revision Notes

A compact review of NLL, cross-entropy, perplexity, bits, leakage, and evaluation limits.

Exercises

Chapter-level practice for language-model evaluation quantities.

Hints

Low-spoiler nudges for the Chapter 8 exercises.

Solutions

Explained solutions for the Chapter 8 exercises.

Before moving on

Compute a tiny cross-entropy example.
Explain perplexity as an average branching-factor intuition.
Use validation text without confusing it for final truth.

Where this leads

Embeddings in Language
Neural Language Models