Introduction

A language model answers a narrow question:

given this context, what can come next?

The answer is not a single word chosen with certainty. It is a probability distribution over possible next tokens.

This is a small statement with large consequences. To predict the next token well, a model must use regularities in the data: spelling, phrases, grammar, facts written in text, code structure, document format, and long dependencies. The objective is simple to state. The work required to do it well is not simple.

A Tiny First Example

Suppose the text seen during training contains these three lines:

I like tea
I like coffee
I like tea

If the context is:

I like

then the observed next tokens are:

tea, coffee, tea

From this tiny corpus, tea is more likely than coffee after the context I like. That is the beginning of language modeling: observe context, count or learn what follows, and produce probabilities.

What This Chapter Deliberately Avoids

This chapter does not yet depend on tokenizers, neural networks, recurrence, attention, or Transformers. Those ideas matter later. Here, the basic prediction problem must become visible first.

By the end of the chapter, you should be able to:

• separate a sequence from a context;
• identify the token being predicted;
• compute a small next-token probability from counts;
• read a model output as a distribution;
• explain why probability is not the same as truth or grammatical approval;
• explain why prediction can force a model to represent structure.

Before Starting

Keep one question in mind:

What information is available before the prediction?

Most mistakes in this chapter come from accidentally letting the model see the answer it is supposed to predict.