The Residual Stream Receives Two Updates

A Transformer block preserves a model-width residual stream while attention and the position-wise MLP add separate updates. Trace both additions numerically and use branch ablations without confusing them with retraining.

The residual stream is the model-width representation passed from one block to the next. Attention and the MLP do not replace it. Each branch calculates an update with the same outer shape and adds that update to the stream.

For the pre-norm convention,

a=\operatorname{Attention}(N_1(x)),\qquad u=x+a,

m=\operatorname{MLP}(N_2(u)),\qquad y=u+m.

The stream follows $x\to u\to y$ . The branch values $a$ and $m$ are temporary updates, not three interchangeable names for the same representation.

Trace Two Positions

Let $B=1$ , $T=2$ , and $d_{model}=4$ . Suppose verified branches return

x= \begin{bmatrix} 1&0&-1&2\\ 0&1&2&-1 \end{bmatrix},

a= \begin{bmatrix} 0.2&-0.1&0.3&0\\ 0.1&0.2&-0.2&0.4 \end{bmatrix}.

The first residual update is

u=x+a= \begin{bmatrix} 1.2&-0.1&-0.7&2\\ 0.1&1.2&1.8&-0.6 \end{bmatrix}.

If the MLP update is

m= \begin{bmatrix} -0.2&0.4&0.1&0.2\\ 0.3&-0.1&0.2&0.1 \end{bmatrix},

then

y=u+m= \begin{bmatrix} 1.0&0.3&-0.6&2.2\\ 0.4&1.1&2.0&-0.5 \end{bmatrix}.

Identity Path and Learned Update

The addition gives each stage a direct identity contribution. If one branch returns zero, that stage passes its input unchanged. This offers an easier function for a branch to approximate than rebuilding the entire representation from scratch.

Residual connections were developed and tested before Transformers. Their successful use in deep residual networks is relevant history and optimization evidence, not a proof that every residual architecture trains reliably under every initialization and optimizer.

Ablate One Branch at a Time

Set $a=0$ to remove the current attention update while keeping the MLP input consistent with the changed $u=x$ .
Set $m=0$ to remove the current MLP update while retaining attention.
Multiply one branch by a scalar $\alpha$ to inspect sensitivity.
Do not compare a removed branch with a separately retrained model as if they were the same intervention.

Changing attention also changes the input later normalized for the MLP. The effect of the attention branch is therefore not limited to adding $a$ once.

Q1. Complete two residual updates

For one token record, $x=[1,-1]$ , attention returns $a=[0.25,0.5]$ , and the MLP later returns $m=[-0.5,0.25]$ . What is the second coordinate of $y=(x+a)+m$ ?

Your answer

Compute it first, then check your number.

Hint

Add the attention update first, then the MLP update.

Solution

u=[1.25,-0.5]

and

y=[0.75,-0.25]

Reference

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep Residual Learning for Image Recognition, 2015.