Summary and Revision Notes

Core ideas

• A unigram model ignores context.
• A bigram model conditions on one previous token.
• A trigram model conditions on two previous tokens.
• Count-based models estimate conditional probabilities from observed counts.
• A Markov assumption limits how much context the model uses.
• Longer contexts can be useful but create sparse count tables.

Key formula

P(next | context) = count(context, next) / count(context)

Check yourself

• Can you compute a unigram probability from a token count?
• Can you compute a bigram probability by hand?
• Can you explain why trigrams are sparser than bigrams?