Introduction

Modern LLMs are large, but their foundation is still language modeling:

given context, predict what can come next

Scale changes what becomes possible. Better data, larger models, longer contexts, and stronger training systems can produce surprising capabilities. But the basic pretraining objective remains connected to the small examples in this subject.

What this chapter covers

• pretraining objective;
• what scale changes;
• data quality;
• benchmarks and limits;
• hallucination and model behavior.