Learned Projections Define Matching and Content

The query and key projections determine learned matching, while the value projection determines the content carried along that route. Controlled ablations show these roles without assigning unsupported human meanings to individual coordinates.

The input representation $x_i$ does not arrive with separate fields named “what I need,” “how I should be found,” and “what I should send.” The matrices $W_Q$ , $W_K$ , and $W_V$ learn those three projections from data.

For model width $d_{model}$ ,

W_Q,W_K\in\mathbb R^{d_{model}\times d_k},\qquad W_V\in\mathbb R^{d_{model}\times d_v}.

Queries and keys require the same width because their dot product contracts over that axis. Values may use another width because they are combined only after the attention weights have been calculated.

Change One Projection at a Time

Use the two input rows

x_1=[1,0],\qquad x_2=[0,1],

and let $W_Q=W_K=W_V=I$ . The first query has dot products

q_1k_1^\top=1,\qquad q_1k_2^\top=0.

It therefore favors the first value. We can now perform three controlled changes.

Change only $W_Q$

Swap the query coordinates:

W_Q'=\begin{bmatrix}0&1\\1&0\end{bmatrix}.

Now $q_1'=[0,1]$ , so its two dot products become 0 and 1. The match reverses, while the keys and values themselves do not change.

Change only $W_K$

Restoring $W_Q=I$ and applying the same coordinate swap to $W_K$ also reverses which key matches the first query. This change alters how every source position presents itself for matching.

Change only $W_V$

Keep $W_Q=W_K=I$ and let

W_V'=\begin{bmatrix}2&0\\0&-3\end{bmatrix}.

The scores and attention weights remain exactly the same, but the values become $[2,0]$ and $[0,-3]$ . The routed content changes even though the routing pattern does not.

Changed parameter	Directly changes	Does not directly change
$W_Q$	each position's requests	source values
$W_K$	how source positions match	source values
$W_V$	content carried by source positions	attention weights

These descriptions refer to one forward pass while the other parameters are held fixed. During training, all projections may adapt together, so observed effects need not remain isolated.

Matching Coordinates Do Not Have Fixed Names

It is tempting to label a query-key coordinate “subject,” “plural,” or “earlier name.” Such a label is a hypothesis about a trained model, not part of the definition. The basis can change while dot products remain the same.

For example, let $R$ be an orthogonal matrix, so $RR^\top=I$ . Replacing

Q' = QR,\qquad K'=KR

does not change the score matrix:

Q'K'^\top=QRR^\top K^\top=QK^\top.

The individual coordinates rotated, but every query-key score stayed fixed. This simple symmetry shows why a coordinate should not receive a semantic name without additional evidence.

Ablate one projection

Change one matrix at a time. Compare the score row with the values and weighted reading.

Q1. Change content without changing routing

Which single projection can be changed while leaving all query-key scores and attention weights fixed in the current forward pass?

Choose one

The query projection W_Q

The key projection W_K

The value projection W_V

Select one choice, then check.

Hint

The score matrix is

QK^\top/\sqrt{d_k}

Solution

Change

W_V

. It changes

V=XW_V

and therefore the reading

AV

, but it does not enter the score or softmax equations.

Treat Labels as Testable Hypotheses

A projection determines a mathematical role. A trained feature may also have a useful human interpretation, but that claim needs evidence from examples, ablations, or interventions. The equations alone establish matching and content flow—not a semantic dictionary for coordinates.

Change One Projection at a Time

Change only WQW_QWQ​

Change only WKW_KWK​

Change only WVW_VWV​