RMSNorm Removes Re-Centering

RMSNorm scales a token record by its root mean square without first subtracting the record's mean. For $r\in\mathbb{R}^{d_{model}}$,

The usual form has a learned gain $\gamma$ and no learned bias. This is a different operation from LayerNorm, not a faster spelling of it.

Calculate the Same Row Again

For $r=[1,2,3,4]$, the mean squared value is

Ignoring epsilon for the hand calculation, the root mean square is $\sqrt{7.5}\approx2.738613$. With $\gamma=1$, RMSNorm returns approximately

Unlike the LayerNorm result for the same row, these coordinates do not have mean zero. Their mean squared value is 1, up to epsilon and rounding.

A Constant Shift Separates the Operations

Add 10 to every coordinate. LayerNorm removes that shared shift when it subtracts the new mean, so its normalized result is unchanged before gain and bias. RMSNorm does not remove the shift, so its normalized direction changes.

Both operations reduce sensitivity to a common rescaling when epsilon is small, but neither makes a full network invariant to arbitrary changes in its inputs. Learned gains, biases elsewhere, nonlinearities, and finite epsilon still matter.

Axis and Parameter Audit

Like LayerNorm in a Transformer, RMSNorm acts independently on the final feature axis of each $(b,t)$ record. Under the usual gain-only convention it has $d_{model}$ learned entries, compared with $2d_{model}$ for LayerNorm with gain and bias.

Reference

• Biao Zhang and Rico Sennrich, Root Mean Square Layer Normalization, 2019.