Conclusion

A language model estimates what can come next.

This chapter used small examples, but the shape is the same in larger systems:

• text is treated as a sequence;
• the model sees a context;
• the target token is not part of that context;
• the model assigns probabilities to possible next tokens;
• the answer is a distribution, not one certain continuation;
• probability is not the same as truth, usefulness, or grammatical approval;
• prediction can reward representations of useful structure.

What We Have Not Added Yet

We have not yet added tokenizers, neural networks, recurrence, attention, or Transformers. That is intentional. The basic prediction problem should be clear before machinery appears.

The next chapter asks what raw text is and how it becomes data. That step matters because a model can only predict over the units and examples we define for it.