Attention Weights Show Routing, Not the Whole Explanation

Inspect every head's attention row while retaining value vectors, output projections, residual paths, MLPs, and later layers. Calculate one weighted value and connect attention patterns to bounded interventions.

An attention weight answers a narrow question: how strongly does this query mix each value record in this head? It does not show the content of those values, the head's output projection, the residual path, the MLP update, or later computation.

Inspect the Four Final-Query Rows

For <bos> X Y, the final position has these recorded weights:

Layer	Head	`<bos>`	`X`	current `Y`
1	1	0.5068	0.1122	0.3810
1	2	0.2047	0.4141	0.3812
2	1	0.1298	0.7685	0.1017
2	2	0.3472	0.4497	0.2031

Layer 2 head 1 places most weight on X. This is useful routing evidence, but “the head copies X” is not yet established.

Weights Mix Values

For a hand-checkable head, suppose

a=[0.5,0.1,0.4]

and its value rows are

v_0=[1,0],\quad v_1=[0,2],\quad v_2=[-1,1].

The mixed value is

0.5v_0+0.1v_1+0.4v_2=[0.1,0.6].

Changing the value vectors changes the result even if the visible weight row stays fixed. The head's slice of $W_O$ then maps this two-value result back to the four-dimensional residual stream.

Test Effect, Not Just Appearance

Zeroing layer 2 head 1 changes the clean X-minus-Y margin from 6.4618 to -2.4767. That intervention shows that this head's normal write matters under the chosen zero replacement. It still does not show that the weight on X alone causes the effect; the Q/K routing and V/O content path act together.

Q1. Calculate a weighted value

Using the weights and values above, what mixed vector is passed to the output projection?

Your answer

Compute it first, then check your number.

Hint

Calculate the two coordinates separately.

Solution

The first coordinate is

0.5-0.4=0.1

; the second is

0.2+0.4=0.6

What the Research Debate Establishes

Jain and Wallace (2019) found weak or non-unique relationships between attention distributions and prediction-based importance measures in specified NLP models. Wiegreffe and Pinter (2019) argued that the conclusion depends on the definition of explanation and the tests used. Together they support careful, test-specific claims—not the slogans “attention explains everything” or “attention can never be informative.”