Chapter 9

Inspecting and Interpreting Transformers

Inspect attention, residual updates, activations, and logits for a specific prediction, then change the computation through an ablation or causal intervention. The chapter separates observation, causal evidence, interpretation, and remaining uncertainty.

A Transformer exposes many internal values: attention weights, residual records, MLP activations, and intermediate logits. These values can help answer a precise question, but none is a complete explanation by itself.

This chapter inspects the trained 368-parameter decoder from Chapters 7 and 8. It compares two three-token prompts:

  • clean: <bos> X Y, where the model strongly predicts X;
  • corrupted: <bos> Y X, where the model strongly predicts Y.

Both prompts are valid corpus sequences. Here, corrupted does not mean malformed or out of distribution. It names the comparison run whose token order removes the clean run's evidence for the chosen X-over-Y question.

The main measurement is the final-position logit margin

m=logit⁡(X)−logit⁡(Y).m=\operatorname{logit}(X)-\operatorname{logit}(Y).

It is 6.4618 for the clean prompt and -5.7350 for the corrupted prompt. That single number makes interventions easy to compare, while the complete output probabilities remain part of the record.

You will learn to:

  • separate observation, association, intervention, and mechanistic evidence;
  • verify that an instrumented forward pass still matches the original model;
  • read attention patterns without treating them as complete explanations;
  • inspect attention and MLP writes to the residual stream;
  • project intermediate residual records into vocabulary space while stating the limits of this diagnostic;
  • ablate heads and branches with an explicit replacement rule;
  • patch activations from a clean run into a matched corrupted run;
  • distinguish localization, necessity, sufficiency, and a complete mechanism;
  • use neuron, feature, polysemanticity, and circuit vocabulary carefully;
  • write a reproducible interpretation report with alternative explanations.

The examples establish facts about this small model, these prompts, and these interventions. They do not imply that every larger language model uses the same components or algorithm.

Chapter 10 will step back from one inspected model and map architectural variants, scaling evidence, and the boundaries of later subjects.

After this chapter

  • Inspect attention, residual updates, activations, and vocabulary logits for one prediction.
  • Distinguish observation, ablation, and causal intervention.
  • Write an interpretation that states alternatives and unknowns.

Lessons

  1. 01
    Start with a Question and an Evidence Ladder

    Frame Transformer interpretation with a precise question, metric, evidence ladder, intervention scope, and bounded claim.

    1 exercise
  2. 02
    Record One Prediction before Explaining It

    Build a reproducible Transformer activation trace with explicit prompts, probabilities, tensor sites, shapes, and forward-pass equivalence.

    1 exercise
  3. 03
    Attention Weights Show Routing, Not the Whole Explanation

    Interpret Transformer attention weights as routing within QK and value-output computation, calculate weighted values, and understand explanation limits.

    1 exercise
  4. 04
    Residual and MLP Writes Change the Running Record

    Inspect Transformer residual-stream, attention-write, MLP-activation, and MLP-write vectors and interpret their norms carefully.

    1 exercise
  5. 05
    Intermediate Vocabulary Projections Are Diagnostic Views

    Use logit-lens-style intermediate vocabulary projections to inspect Transformer residual states while preserving calibration and causal limits.

    1 exercise
  6. 06
    Ablation Tests Dependence on a Chosen Replacement

    Perform Transformer head and branch ablations, calculate logit-margin effects, and understand replacement, interaction, necessity, and sufficiency limits.

    1 exercise
  7. 07
    Activation Patching Compares Matched Runs

    Perform clean-to-corrupted Transformer activation patching, calculate recovery, name exact sites, and interpret localization controls and limits.

    1 exercise
  8. 08
    Localization Is Not Yet a Complete Mechanism

    Separate Transformer component localization from a complete mechanism using necessity, sufficiency, specificity, interactions, and controls.

    1 exercise
  9. 09
    Neurons, Features, Polysemanticity, and Circuits

    Learn mechanistic interpretability vocabulary for Transformer neurons, activations, features, polysemanticity, superposition, and circuits.

    1 exercise
  10. 10
    Write a Reproducible Interpretation Report

    Write and audit a reproducible Transformer interpretation report with experiments, controls, claim levels, alternatives, and next tests.

    1 exercise

Review and practice

  1. •
    Review

    Review Transformer inspection and interpretation through trace verification, attention, residuals, logits, ablations, activation patches, and bounded claims.

  2. •
    Exercises

    Solve Transformer interpretability exercises on evidence, tensors, attention, residuals, logits, ablation, patching, circuits, and reporting.

Chapter progress