Backoff

Backoff uses a shorter context when the longer context has too little evidence.

For example:

try trigram context
if unreliable, use bigram context
if unreliable, use unigram context

The intuition is plain: use the most specific evidence when it exists, but do not fail completely when it does not.

Example

If I really like -> tea was never observed, the model can back off to like -> tea. If that is also missing, it can use the general frequency of tea.

Backoff is not the same as interpolation. Backoff moves to a lower-order model when needed. Interpolation mixes several orders.

Exercise

In a backoff chain from trigram to bigram to unigram, how many model orders are available?