Gradient Norms

Gradient norms summarize gradient size.

A norm turns many gradient values into one magnitude. It does not show every detail, but it helps reveal whether update signals are tiny, huge, or uneven across layers.

Gradient norms can reveal layers with unusually large or tiny update signals.

If gradients are near zero, the model may not update meaningfully. If gradients are extremely large, training may be unstable. If one layer has a much larger norm than others, inspect that layer and the operations around it.

For a vector gradient [3, 4], the L2 norm is:

sqrt(3^2 + 4^2) = 5

Exercise: Gradient norm

What is the L2 norm of gradient vector [6, 8]?

Your answer

Compute it first, then check your number.

Exercise: Tiny gradient signal

Enter 1 for likely healthy update signal, or 2 for warning: all gradient norms are near 0 for many steps.

Your answer

Compute it first, then check your number.