LayerNorm Normalizes Each Token Record

LayerNorm calculates mean and population variance across one token's feature coordinates, then applies learned gain and bias. Work the calculation by hand and test that tokens and batch items remain independent.

Layer normalization calculates statistics across the feature coordinates of one token record. For $r\in\mathbb{R}^{d_{model}}$ ,

\mu=\frac{1}{d_{model}}\sum_{k=1}^{d_{model}}r_k,

\sigma^2=\frac{1}{d_{model}} \sum_{k=1}^{d_{model}}(r_k-\mu)^2,

\operatorname{LayerNorm}(r)_k =\gamma_k\frac{r_k-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_k.

The variance here uses denominator $d_{model}$ , not the sample-variance denominator $d_{model}-1$ .

Calculate One Row

Let

r=[1,2,3,4].

Its mean is $2.5$ . The squared centered values are

[2.25,0.25,0.25,2.25],

so the population variance is

\sigma^2=5/4=1.25.

Ignoring epsilon only for this hand calculation and using $\gamma=1$ , $\beta=0$ gives approximately

[-1.341641,-0.447214,0.447214,1.341641].

The normalized coordinates have mean 0 and mean squared value 1, up to epsilon and rounding.

Gain and Bias Restore Learned Freedom

$\gamma$ and $\beta$ each have shape $(d_{model})$ . They scale and shift individual normalized features. If the first gain is 2 while the other gains remain 1, only the first normalized coordinate doubles.

LayerNorm therefore adds $2d_{model}$ learned entries under the usual gain-plus-bias convention. Some implementations omit one of them; parameter counts must name the convention.

The Axis Is the Last Feature Axis

For $R:(B,T,d_{model})$ , LayerNorm calculates a separate $\mu$ and $\sigma^2$ for every $(b,t)$ record. It does not combine:

two sequence positions;
two examples in the batch;
statistics accumulated from earlier batches.

Changing one record must not change another record's LayerNorm output. A wrong-axis implementation may keep the same outer shape while coupling tokens or batch items.

Audit LayerNorm records independently

The implementation returns statistics for every row and checks that changing one row leaves another row unchanged.

Q1. Calculate LayerNorm variance

For $r=[1,2,3,4]$ , what population variance does LayerNorm use before adding epsilon?

Your answer

Compute it first, then check your number.

Hint

The mean is 2.5. Divide the sum of squared deviations by 4.

Solution

((-1.5)^2+(-0.5)^2+0.5^2+1.5^2)/4=5/4=1.25

Reference

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton, Layer Normalization, 2016.