Pre-Norm and Post-Norm Change Computation Order

The location of normalization changes the function calculated by a residual branch. For a branch $F$, pre-norm uses

whereas post-norm uses

The same words—normalization, branch, residual addition—can therefore describe two different computation graphs.

Calculate Both Orders

Take $x=[1,3]$, let $F(z)=0.5z$, and use LayerNorm with unit gain, zero bias, and epsilon omitted for this hand calculation. Then $N(x)=[-1,1]$.

Pre-norm gives

Post-norm first gives $x+F(x)=[1.5,4.5]$ and then normalizes it:

The outputs differ even though $x$, $F$, and $N$ are unchanged.

A Complete Block Has Two Placements

This subject implements the pre-norm block

The original Transformer used post-norm residual sublayers. Later work studied why pre-norm variants often behave differently at initialization and during optimization. These results are useful evidence about specific assumptions; they do not establish one placement as universally best.

State the Convention Before Comparing Models

A reproducible architecture description should name:

• pre-norm or post-norm placement;
• LayerNorm, RMSNorm, or another operation;
• whether gain and bias are learned;
• epsilon and the axis being normalized;
• where dropout is applied relative to the branch and addition.

Without these details, two implementations called “the same Transformer block” may calculate different functions.

References

• Ashish Vaswani et al., Attention Is All You Need, 2017.
• Ruibin Xiong et al., On Layer Normalization in the Transformer Architecture, 2020.