Residual and MLP Writes Change the Running Record

Trace attention and MLP additions to the residual stream and compare their norms without confusing magnitude with direction, output alignment, or causal importance. Separate MLP hidden activations from their writes.

Each pre-normalized decoder block adds two writes to the residual stream:

r^{\prime}=r+a,\qquad r^{next}=r^{\prime}+u,

where $a$ is the attention write and $u$ is the MLP write. Recording all three vectors shows what changed at a position without pretending that the residual stream performs a separate hidden operation.

Compare Magnitudes at the Final Position

For the clean prompt:

Layer	$\lVert r\rVert_2$ before	$\lVert a\rVert_2$	$\lVert u\rVert_2$	$\lVert r^{next}\rVert_2$
1	2.5121	0.2425	0.9511	2.1260
2	2.1260	1.5302	0.4572	1.5367

Layer 2's attention write has a larger norm than its MLP write. This does not prove that attention is more important. Norm measures magnitude, not direction, alignment with the output readout, redundancy with another component, or effect under intervention.

Vector Addition Can Reduce a Norm

The final residual norm after layer 2 is smaller than the incoming norm even though nonzero vectors were added. If an update points partly against the current residual, cancellation reduces the resulting magnitude.

For example:

r=[1,-1],\quad a=[0.5,0.5],\quad u=[-0.25,1],

r+a+u=[1.25,0.5].

The component norms alone cannot reconstruct this direction.

MLP Activations Are Not MLP Writes

For one position, the first MLP layer creates eight preactivations, ReLU keeps positive entries, and the second linear layer maps them back to four residual features:

u=W_2\operatorname{ReLU}(W_1\operatorname{LN}(r^{\prime})+b_1)+b_2.

A large hidden activation may have a small, opposing, or shared output effect depending on its column in $W_2$ and the other active units.

Q1. Interpret a component norm

Layer 2 has attention-write norm 1.5302 and MLP-write norm 0.4572. What may you conclude without another test?

Choose one

The attention write has the larger L2 magnitude at this position.

Attention is the more important causal mechanism for this prediction.

The attention write represents more semantic information than the MLP write.

Select one choice, then check.

Hint

Do not turn vector length into a claim about direction or output effect.

Solution

The attention write has the larger recorded

L_2

magnitude at this position. An output metric or intervention is needed for an importance claim.

The Residual Stream Is a Useful Accounting Interface

Elhage et al. (2021) develop the view that Transformer components read from and write to a shared residual stream. Their circuit framework is valuable vocabulary for additive paths and head composition, but its attention-only simplifications and research status must remain visible when applying it to this decoder with MLPs and normalization.