Introduction

A language model should assign high probability to text that is likely under the data distribution. Evaluation asks whether it does.

Accuracy is not enough for this job. A language model returns probabilities, not only right-or-wrong labels. We need measurements that reward confident good predictions and punish confident bad predictions.

This chapter introduces negative log likelihood, cross-entropy, perplexity, bits, and the limits of evaluation.

What this chapter covers

• negative log likelihood;
• cross-entropy;
• perplexity;
• bits per token and byte;
• memorization, leakage, and evaluation limits.