Summary and Revision Notes

Core Idea

A language model maps context to a probability distribution over possible next tokens.

context -> next-token probabilities

Terms

Term	Meaning
sequence	an ordered list of items
context	the items the model can see before prediction
target	the token the model is trying to predict
token	the unit being predicted
continuation	a possible next token or next span
distribution	probabilities assigned across possibilities

Small Count Example

Training text:

I like tea
I like coffee
I like tea

For context I like:

P(\text{tea} \mid \text{I like}) = 2/3

P(\text{coffee} \mid \text{I like}) = 1/3

Important Cautions

A language model is not mainly a grammar checker.
The most likely token is not the only possible token.
Probability is not the same as truth.
Prediction can learn structure, but the objective alone does not guarantee useful behavior.
A generated token is a use of the distribution, not the distribution itself.

Before Moving On

You should be able to explain what a language model predicts, compute a tiny next-token probability from counts, and say why the output is a distribution.