Chapter 6

Decoder-Only Language Models

Trace a complete causal Transformer from token IDs through embeddings, decoder blocks, final normalization, vocabulary logits, shifted targets, and cross-entropy. Every tensor and parameter remains visible in one tiny model.

A decoder-only language model turns integer token IDs into one vocabulary-score row at every sequence position. Between those endpoints, token and position records enter causal Transformer blocks, a final normalization prepares the readout, and an unembedding maps model width to vocabulary width.

This chapter uses one frozen toy model throughout:

QuantityValue
vocabulary size8
maximum context4
model width4
decoder blocks2
attention heads2
head width2
MLP width8

It uses learned absolute positions, pre-LayerNorm blocks, ReLU MLPs, a final LayerNorm, no dropout in deterministic audits, and tied token embedding and unembedding weights. These choices make a complete model inspectable; they are not claims about universal production defaults.

For IDs I:(B,T)I:(B,T) with T≤4T\le4, the forward path is

I→E[I]+P[:T]→Block⁡1→Block⁡2→Nf→Z.I\rightarrow E[I]+P[:T]\rightarrow \operatorname{Block}_1\rightarrow\operatorname{Block}_2 \rightarrow N_f\rightarrow Z.

The logits have shape Z:(B,T,8)Z:(B,T,8). Each valid row is aligned with exactly one next-token target before cross-entropy is averaged.

After this chapter

  • Trace token IDs through a complete decoder-only forward pass.
  • Explain final normalization, unembedding, weight tying, logits, and shifted targets.
  • Calculate tensor shapes, parameter counts, and token-level loss.

Lessons

  1. 01
    Token IDs Become Position-Aware Records

    Trace token embedding lookup, learned position addition, tensor shapes, parameter counts, and input-range validation.

    1 exercise
  2. 02
    Causal Decoder Blocks Update Every Prefix Position

    Trace repeated causal decoder blocks, residual-stream shapes, masks at every layer, distinct parameters, and per-block counts.

    1 exercise
  3. 03
    Final Normalization Prepares the Vocabulary Readout

    Calculate decoder final LayerNorm, its feature axis, learned parameters, independence, and role before unembedding.

    1 exercise
  4. 04
    Unembedding Produces One Logit per Vocabulary Token

    Derive decoder unembedding shapes, calculate tied vocabulary logits, and distinguish stable vocabulary softmax from other axes.

    1 exercise
  5. 05
    Weight Tying Reuses the Token Embedding Matrix

    Derive input-output embedding weight tying, transpose shapes, shared identity tests, and tied-versus-untied parameter counts.

    1 exercise
  6. 06
    Shifted Targets Turn Logits into Next-Token Loss

    Trace shifted decoder targets, stable log-sum-exp cross-entropy, valid-target means, and common alignment failures.

    1 exercise
  7. 07
    Trace the Complete Tiny Decoder

    Implement and audit the frozen tiny decoder from IDs through causal blocks, tied logits, probabilities, loss, and causality tests.

    1 exercise
  8. 08
    Audit Masks, Targets, Vocabulary, and Readout

    Debug decoder vocabulary, context, causal masks, targets, axes, weight tying, loss masking, and parameter counts.

    1 exercise

Review and practice

  1. •
    Review

    Review the complete tiny decoder equations, configuration, parameters, targets, loss, and debugging checks.

  2. •
    Exercises

    Solve decoder-only Transformer lookup, shape, parameter, logit, loss, causality, tying, and debugging exercises.

Chapter progress