Conditional Probability

Count-based language models estimate conditional probabilities.

For a bigram model:

P(next token | previous token)

The estimate is:

count(previous token, next token) / count(previous token)

Suppose:

count(I, like) = 4
count(I) = 5

Then:

P(like | I) = 4 / 5

This is the same probability idea from Mathematics, applied to text contexts.

Exercise

If count(the, cat) = 6 and count(the) = 10, what is 10 * P(cat | the)?