Next-Token Prediction

Next-token prediction asks for the next item in a sequence.

Suppose the training text contains these short sequences:

I like tea
I like coffee
I like tea

For the context I like, the next-token counts are:

next token	count
tea	2
coffee	1

There are 3 observations after the context. A tiny count-based predictor can turn the counts into probabilities:

P(\text{tea} \mid \text{I like}) = \frac{2}{3}

P(\text{coffee} \mid \text{I like}) = \frac{1}{3}

The model is not saying coffee is impossible. It is saying tea appeared more often after this context in this tiny corpus.

The General Shape

The basic language-modeling shape is:

context -> possible next tokens -> probabilities

Count tables make this visible. Neural language models later replace the table with learned parameters, but the prediction question remains the same.

LM-C01-T02-001Exercise: Count the next token

In the examples above, how many times does tea appear after I like?

Your answer

Compute it first, then check your number.

Hint

Look only at examples whose context is exactly I like.

Solution

The examples after I like are tea, coffee, and tea. The token tea appears twice, so the count is 2.

LM-C01-T02-002Exercise: Compute the probability

Using the same counts, what is P(coffee | I like) as a decimal?

Your answer

Compute it first, then check your number.

Hint

Use count divided by total observations after the same context.

Solution

coffee appears once after I like. There are 3 total next-token observations after that context.

P(\text{coffee} \mid \text{I like}) = \frac{1}{3} \approx 0.33

Prediction begins with a conditional question: given this context, how should probability be distributed over the next token?