Conclusion

Smoothing and backoff solve a practical problem in count-based language modeling: the training corpus is incomplete.

An unseen event should not always mean an impossible event. Add-one smoothing, interpolation, and backoff are simple ways to see that principle. Good-Turing and Kneser-Ney show that older language modeling already contained serious work on rare events and missing evidence.

Next we need a way to measure whether a language model is good. That brings us to evaluation, cross-entropy, and perplexity.