Introduction

Plain count models have a sharp failure: unseen events get probability zero.

That is too harsh. If a corpus never contains green telescope, it does not mean the phrase is impossible. It may only mean the corpus is small.

Smoothing and backoff are ways to manage missing evidence. They do not make the model intelligent. They make the probability estimates less brittle.

What this chapter covers

• the zero-probability problem;
• add-one smoothing;
• interpolation;
• backoff;
• Good-Turing and Kneser-Ney as historical ideas.