The MLP Mixes Features at Each Position

After attention exchanges information across positions, the block applies the same multilayer perceptron to every token record independently. A basic two-projection MLP is

For model width $d_{model}$ and hidden width $d_{ff}$,

The MLP expands the feature axis and contracts it again. It does not change sequence length.

Trace Two Positions Independently

Let $d_{model}=2$, $d_{ff}=3$, use ReLU, omit biases, and choose

For $r_1=[1,2]$, the expanded values are $[1,2,-1]$. ReLU gives $[1,2,0]$, so the output is $[1,2]$.

For $r_2=[-1,1]$, the expanded values are $[-1,1,-2]$. ReLU gives $[0,1,0]$, so the output is $[0,1]$.

The weights are shared, but each position follows its own calculation.

Position-Wise Does Not Mean Feature-Wise

Changing one token record must not alter another token's MLP output. Within one record, however, each hidden unit can combine every input feature, and each output coordinate can combine every hidden unit. The MLP is independent across positions but dense across features.

This gives a useful implementation test: copy an input sequence, alter one position, run only the MLP, and confirm that all other output positions remain unchanged.

Count the Learned Entries

With two biases, the parameter count is

For $d_{model}=2$ and $d_{ff}=3$, this is $6+3+6+2=17$. A count that omits biases must say so.