Decoder-Only Models Predict the Next Token

A decoder-only language model uses one causal token stream. At each position, it produces vocabulary logits for the next token while preventing information from later tokens from entering the current representation.

Make the Shift Visible

For the token sequence

A B C D

one training window can be written as:

The input tensor may contain A B C, with target tensor B C D. A causal mask ensures that output row 1 cannot read input B or C, even though all three input positions are processed in one parallel training pass.

Parallel Training Does Not Remove Causality

During training, the complete known sequence permits calculation of all positions at once. The mask preserves the conditional factorization:

During generation, B is not available until it has been selected, so outputs are produced one token at a time. Training parallelism and generation order are different properties.

The Diagonal Depends on the Shift Convention

It is common for representation row $t$ to read input token $t$ and predict token $t+1$. Under that convention the causal mask includes its diagonal. Other indexing or shifted-input descriptions can look different while encoding the same conditional problem. State tokens, rows, and targets rather than arguing from “strict” versus “inclusive” triangles alone.

Decoder-Only Does Not Mean Chat Model

The base architecture and next-token objective do not by themselves define an instruction-following assistant. Data selection, scale, post-training, decoding, retrieval or tools, and system policies alter observed behavior. This chapter uses GPT as an early representative of generative Transformer pretraining, not as a definition of every later LLM system.

Reference

• Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, Improving Language Understanding by Generative Pre-Training, 2018.