The Output Projection Restores the Model Width

The weighted values may have width d_v, so W_O maps them back to the model width expected by the rest of the network. Calculate this projection, count its parameters, and test identity, zero, and learned alternatives.

The weighted values form

H=AV\in\mathbb R^{T\times d_v}.

This is a valid attention result, but it does not yet have the promised layer interface when $d_v\ne d_{model}$ . The output projection completes the layer:

Y=HW_O,\qquad W_O\in\mathbb R^{d_v\times d_{model}},\qquad Y\in\mathbb R^{T\times d_{model}}.

The operation changes feature coordinates independently at every position. It does not mix positions; that mixing already happened in $AV$ .

Continue the Numerical Trace

The final row of the four-token attention trace was

h_4\approx[0.592,-1.279].

Choose a non-identity output projection

W_O=\begin{bmatrix} 1&0.5\\ -1&2 \end{bmatrix}.

Then

\begin{aligned} y_4 &=h_4W_O\\ &\approx[0.592,-1.279] \begin{bmatrix}1&0.5\\-1&2\end{bmatrix}\\ &\approx[1.871,-2.262]. \end{aligned}

The attention weights did not change. $W_O$ changed how the resulting value features were expressed in model-width coordinates.

Three Ablations Clarify Its Role

An ablation removes or changes one component while holding the rest of the setup fixed. Comparing the resulting computation helps identify what that component contributes.

Output projection	Result	What the comparison shows
$W_O=I$ when widths match	$Y=H$	no feature remapping
$W_O=0$	$Y=0$	this attention layer writes no signal onward
learned non-identity $W_O$	$Y$ mixes value features	the layer can choose how attention output is expressed

The zero ablation does not prove that a trained model never needed attention; in a complete network, other residual paths may partially compensate. It does give an exact statement about this forward pass: the attention branch contributes a zero update.

Count the Parameters

Without bias vectors, a single head contains

2d_{model}d_k+d_{model}d_v+d_vd_{model}

parameters: one query matrix, one key matrix, one value matrix, and one output matrix. If $d_k=d_v=d_{model}=d$ , the count is $4d^2$ .

Some implementations add bias vectors to one or more projections. The formulas in this subject omit them unless a lesson states otherwise. A parameter count is meaningful only after such conventions are explicit.

Projection Cannot Recover Discarded Information

$W_O$ can remix the coordinates in $H$ , but it cannot recover separate source values after they have been combined into the same weighted sum. If two different sets of values produce the same $H$ , multiplying both by the same $W_O$ still produces the same $Y$ .

This limit matters when interpreting attention. The output projection can change or cancel features, but it cannot reveal which source decomposition was the uniquely intended one when the weighted sum itself is ambiguous.

Q1. Project one attention reading

Let $h=[2,-1]$ and

W_O=\begin{bmatrix}1&3\\2&0\end{bmatrix}.

What is the second coordinate of $y=hW_O$ ?

Your answer

Compute it first, then check your number.

Hint

Use the second column of

W_O

Solution

The second coordinate is

2(3)+(-1)(0)=6

. The complete output is

[0,6]

Preserve the Layer Contract

When checking an attention implementation, do not stop at the weighted values. Verify that the output projection returns the final feature axis to $d_{model}$ . That invariant is what allows later Transformer blocks to reuse one model width across many residual updates.