Why Scale Matters

Neural networks repeatedly multiply, add, and apply nonlinear functions. Repetition makes scale important.

If each layer roughly doubles the size of its values, five layers can multiply scale by about 2^5 = 32. If each layer halves the scale, five layers can shrink it by 1 / 32.

This affects both activations and gradients. Large activations can produce unstable losses. Tiny activations can make layers nearly silent. Large gradients can create destructive updates. Tiny gradients can make training crawl.

Scale is not only about avoiding numerical overflow. It is about keeping information visible.

good scale: values change enough to matter
bad scale: values explode, vanish, saturate, or become indistinguishable