Good-Turing and Kneser-Ney

Two smoothing names are worth recognizing: Good-Turing and Kneser-Ney.

You do not need the full derivations here. The important point is historical and conceptual: strong count-based language modeling required careful treatment of rare and unseen events.

Good-Turing

Good-Turing smoothing asks how much probability mass should be reserved for things not yet seen. It uses the pattern of rare events to reason about unseen events.

Kneser-Ney

Kneser-Ney smoothing became important for n-gram language models because it does more than add a small constant. It cares about how widely a word appears across contexts.

For example, a word that follows many different contexts should be treated differently from a word that appears often only after one phrase.

Exercise

How many historical smoothing names are introduced on this page?