Chapter 1

From Attention to Self-Attention

Turn the single causal attention operation from Language Modeling into a reusable self-attention layer. The chapter adds the output projection and batched matrix view while keeping every axis, mask, weight, and intermediate value inspectable.

Language Modeling ended with a complete causal attention calculation. A query was compared with keys, a mask removed future positions, softmax produced weights, and those weights combined values. That calculation is the central operation in this chapter, but it is not yet a layer that can be placed inside a Transformer.

A reusable layer needs a clear interface. It receives token representations with model width dmodeld_{model} and must return one representation per token with the same width. It must also process several sequences as a batch without letting one sequence read another. These requirements add an output projection, a batch axis, and a set of shape and numerical checks.

The lessons first distinguish one attention reading from a reusable layer and then examine where queries, keys, and values come from. They assemble the matrix calculation, restore model width with the output projection, add the batch axis, and finish by auditing one small implementation in dependency order.

If the attention calculation itself is unfamiliar, the focused refresher is Queries Compare with Keys and Read Values. The complete numerical trace appears in Attention Returns a Weighted Vector Sum. This chapter extends that trace rather than repeating its full derivation.

After this chapter

  • Distinguish self-attention from cross-attention.
  • Trace Q, K, V, masks, scores, weights, values, and output projection with explicit shapes.
  • Audit one complete single-head self-attention layer in matrix and batched form.

Lessons

  1. 01
    One Attention Reading Is Not Yet a Transformer Layer

    Separate one attention calculation from a reusable self-attention layer with model-width output, batching, and an explicit architectural boundary.

    1 exercise
  2. 02
    One Sequence Supplies Queries, Keys, and Values

    Define self-attention by the source of Q, K, and V, distinguish causal visibility from source choice, and compare cross-attention.

    1 exercise
  3. 03
    Learned Projections Define Matching and Content

    Ablate W_Q, W_K, and W_V separately, trace their effects, and distinguish mathematical roles from semantic interpretation claims.

    1 exercise
  4. 04
    Self-Attention in Matrix Form

    Assemble single-head self-attention in matrix form with explicit axes, causal masking, stable softmax, and the four-token trace.

    1 exercise
  5. 05
    The Output Projection Restores the Model Width

    Derive and calculate W_O, restore d_model, count single-head projection parameters, and interpret output-projection ablations.

    1 exercise
  6. 06
    Add the Batch Axis Without Mixing Sequences

    Move from sequence matrices to batch-major tensors, broadcast masks safely, and verify that batch items remain independent.

    1 exercise
  7. 07
    What Self-Attention Changes and What It Costs

    Compare direct information paths with quadratic dense cost, training parallelism with generation order, and attention observations with causal claims.

    1 exercise
  8. 08
    Audit a Single-Head Self-Attention Layer

    Implement and audit one causal self-attention layer with shape, mask, numerical, output, and batching invariants.

    1 exercise

Review and practice

  1. •
    Review

    Review the self-attention equation chain, distinctions, ablations, costs, limitations, and debugging checklist.

  2. •
    Exercises

    Solve shape, calculation, masking, projection, parameter-count, debugging, batching, and interpretation exercises for self-attention.

Chapter progress