Summary and Revision Notes

Core ideas

• Plain count models assign zero probability to unseen continuations.
• Zero probability is often too strong for incomplete data.
• Add-one smoothing adds 1 to every vocabulary item before normalizing.
• Interpolation mixes estimates from several context lengths.
• Backoff uses a shorter context when the longer context lacks evidence.
• Good-Turing and Kneser-Ney are important historical smoothing ideas.

Check yourself

• Can you explain why zero probability is brittle?
• Can you compute an add-one smoothed count?
• Can you distinguish interpolation from backoff?