One Wider Head Still Has One Attention Pattern

A wider value vector carries more features, but every feature in one head still uses the same attention-weight row. Compare this with multiple heads, which can form independently normalized reading patterns without guaranteeing that specialization occurs.

For one head and one query position, the reading is

z_i=\sum_{j=1}^{T}a_{ij}v_j.

The value $v_j$ may be wide. It may have 2 coordinates, 64 coordinates, or the full model width. That width changes how much content each source position can carry, but it does not create another attention distribution. The same scalar $a_{ij}$ multiplies every coordinate of $v_j$ .

Widening Values Does Not Add Routing Choices

Suppose one query assigns weights

a_i=[0.8,0.2]

to two values

v_1=[1,0,10,0],\qquad v_2=[0,1,0,20].

The reading is

z_i=0.8v_1+0.2v_2=[0.8,0.2,8,4].

The final two value coordinates did not receive a new routing pattern. They used the same weights $[0.8,0.2]$ as the first two coordinates.

Now consider two heads. Head 1 may assign $[0.8,0.2]$ , while head 2 assigns $[0.1,0.9]$ . Their value readings can emphasize different positions before they are joined. This is the additional representational possibility created by multiple heads: several independently normalized routing distributions.

Possibility Is Not a Guarantee

Separate projection matrices allow heads to produce different Q, K, V, and attention weights. The definition does not force them to do so. Two heads may learn similar patterns, one may contribute little, or their roles may vary by input and layer.

It is therefore precise to say:

Multiple heads can form different attention distributions in different learned subspaces.

It is too strong to say:

Every head learns one distinct linguistic rule.

The second statement needs evidence from a particular trained model, task, input distribution, and analysis method.

One Head Can Still Mix Many Features

This distinction does not make a single head weak in every sense. Its value and output projections may transform many features, and one weighted sum may carry several signals. The limitation is narrower: for a fixed query, all value features inside that head share one vector of position weights.

Q1. Count routing distributions

A single attention head has value width 128. For one query, how many separately normalized attention distributions does that head form over key positions?

Your answer

Compute it first, then check your number.

Hint

Softmax is applied once to the head's query-key score row.

Solution

One. All 128 value coordinates are combined using the same attention weight row for that query.

Ask What Is Independently Normalized

When comparing architectures, do not infer the number of attention patterns from feature width. Count the independently projected query-key score matrices and their softmax operations. That count is the number of heads.