Localization Is Not Yet a Complete Mechanism
Compare activation reading, prompt association, ablation, and patching by the question each answers. Distinguish localization, necessity, sufficiency, specificity, interactions, controls, and held-out predictions.
Attention inspection, ablation, and activation patching answer related but different questions:
| Method | Direct question | Important limitation |
|---|---|---|
| Read an activation | What value occurred? | Presence is not output influence. |
| Compare prompts | What value changed? | Both may depend on another cause. |
| Zero ablation | What happens under zero replacement? | Zero may be unusual; compensation may hide a role. |
| Clean-to-corrupted patch | Can this clean value restore a metric here? | A broad or late patch can restore many causes at once. |
A layer-position heatmap is a localization result. A mechanism must also state what information is computed, how it moves, which operations transform it, and which tests could disprove the account.
Ask Necessity, Sufficiency, and Specificity Separately
- Necessity: does removing the component damage the behavior under a stated replacement?
- Sufficiency: does inserting or preserving it produce the behavior when other required conditions are controlled?
- Specificity: does the intervention preserve unrelated behaviors or outputs?
- Generalization: does the account predict results on held-out prompts or related tasks?
One experiment rarely answers all four.
Test Interactions
Suppose ablation effects for heads A and B are -3 and -2. Removing both need not give -5. Later nonlinear operations may amplify, suppress, or reroute the combined change. Redundant components can each show a small individual effect but a large joint effect.
Define an interaction term for one metric:
Nonzero shows that the joint intervention is not additive under this metric and replacement. It does not identify the reason by itself.
Use Positive and Negative Controls
The Chapter 9 patch table includes:
- a negative control: patching identical position-0 state produces zero recovery;
- a positive endpoint: patching the complete final residual produces full recovery;
- intermediate sites with partial recovery.
These controls show that the patching code can produce both no effect and the expected maximum effect. They do not validate every interpretation of an intermediate site.
Q1. Identify an incomplete mechanism claim
A final-position residual patch immediately before the readout gives recovery 1.0. Which conclusion is supported?
Select one choice, then check.
Hint
Solution
Prefer Accounts that Predict New Tests
A useful hypothesis says which head should attend where, which value should be moved, which residual direction should change, and what a new prompt or intervention should do. A post-hoc label that fits only the original heatmap is not yet a strong mechanism.