Chapter 7
Smoothing and Backoff
Add-one smoothing, interpolation, backoff, Good-Turing and Kneser-Ney as historical names, rare events, and sparse-data overconfidence.
What this chapter does
Smoothing teaches how simple models handle missing evidence without pretending unseen means impossible.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Smoothing and backoff as ways to handle missing evidence in count models.
- 02The Zero-Probability Problem
Why unseen continuations should not always be treated as impossible.
- 03Add-One Smoothing
Adding one count to each vocabulary item before normalizing.
- 04Interpolation
Mixing estimates from several context lengths.
- 05Backoff
Using a shorter context when the longer context lacks evidence.
- 06Good-Turing and Kneser-Ney
Recognizing historical smoothing ideas for rare and unseen events.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What smoothing and backoff establish before language-model evaluation.
Summary and Revision NotesA compact review of zero probability, add-one smoothing, interpolation, backoff, Good-Turing, and Kneser-Ney.
ExercisesChapter-level practice for smoothing counts and choosing shorter contexts.
HintsLow-spoiler nudges for the Chapter 7 exercises.
SolutionsExplained solutions for the Chapter 7 exercises.
Before moving on
- Explain why unseen phrases should not always receive zero probability.
- Compare smoothed and unsmoothed probabilities.
- Recognize smoothing as evidence management, not a hack.
Where this leads
- Evaluating Language Models
- Neural Language Models