Pretraining Objective

Pretraining teaches a model from large text corpora before it is adapted for specific use.

For autoregressive language models, the central objective is next-token prediction. The model sees previous tokens and learns to assign probability to the next token.

This objective is simple to state, but rich in pressure. To predict well, the model can benefit from syntax, facts, style, code patterns, reasoning traces, and document structure.

Exercise

In next-token pretraining, how many next-token targets does one prediction position have?