Transformers
Build Transformer architectures from inspectable operations. Develop self-attention, multiple heads, position, residual blocks, architecture families, training, generation, caching, and careful internal inspection.
Language Modeling established the prediction problem and derived one causal attention operation. We now turn that operation into a complete architecture: several heads read context, position information represents order, residual blocks update each token representation, and an output layer produces vocabulary logits.
The examples remain small enough to inspect. You will calculate attention rows, follow tensor axes through a block, compare architecture families, train a tiny model, verify cached generation, and test interpretations with controlled interventions. Python, Mathematics, Deep Learning, and Language Modeling provide the prerequisites; the opening chapter links to focused refreshers when they help.
Chapters
The review sequence follows the computation from one self-attention layer through architecture families to a trained and inspected decoder-only Transformer. The family section is being expanded before publication.
Extend one causal attention calculation into a complete single-head self-attention layer with learned projections, batching, explicit shapes, and an output projection.
8 lessons18 exercisesChapter 2Multi-Head AttentionTrace multi-head attention through projections, head splitting, parallel attention, concatenation, output projection, parameter counts, and ablations.
8 lessons18 exercisesChapter 3Position and Sequence OrderLearn how Transformers represent order with absolute, sinusoidal, relative, rotary, and linear-bias position methods.
8 lessons18 exercisesChapter 4Inside a Transformer BlockBuild a Transformer block from residual updates, attention, LayerNorm or RMSNorm, a position-wise MLP, activation functions, and pre-norm or post-norm ordering.
8 lessons18 exercisesChapter 5Transformer Architecture FamiliesDistinguish encoder-only, decoder-only, and encoder-decoder Transformers by attention flow, masks, objectives, and representative uses.
8 lessons18 exercisesChapter 6Decoder-Only Language ModelsBuild and audit a decoder-only Transformer language model from token IDs to vocabulary logits, shifted targets, and next-token loss.
8 lessons18 exercisesChapter 7Training a Tiny TransformerTrain and debug a tiny Transformer with reproducible batches, target alignment, initialization, optimization, validation, checkpoints, and diagnostic plots.
10 lessons20 exercisesChapter 8Autoregressive Generation and Efficient InferenceTrace autoregressive generation, prefill and decode phases, KV-cache construction, context limits, memory growth, and cached-versus-uncached correctness.
10 lessons20 exercisesChapter 9Inspecting and Interpreting TransformersInspect and intervene on a small Transformer with attention views, residual and activation traces, logits, ablations, patching, and bounded mechanistic claims.
10 lessons20 exercisesChapter 10Variants, Scaling, and BoundariesCompare Transformer variants, scaling evidence, long-context and mixture-of-experts previews, multimodal directions, alternatives, and later curriculum boundaries.
10 lessons20 exercisesProjects
The project combines the complete decoder path in one sustained investigation. Its seven milestones have separate progress and reuse the checked model, cache, and interpretation artifacts from the chapters.