Trigram Models

A trigram model uses the previous two tokens as context.

For example:

I like tea
I like code

The context I like is followed by tea once and code once.

So:

P(tea | I like) = 1 / 2
P(code | I like) = 1 / 2

Trigrams can represent more context than bigrams, but the count table becomes larger and sparser.

Exercise

For the context I like above, how many different next tokens are observed?