Gradient Scale

Gradients tell the optimizer how parameters affect the loss. Their scale controls update size.

If gradients are too large, updates can be too large even with a reasonable learning rate. If gradients are too small, updates may barely change the parameters.

The update size is shaped by both gradient and learning rate:

update_size = learning_rate * gradient

But gradient scale also changes across layers. In deep networks, repeated chain-rule products can make earlier-layer gradients shrink or grow. This is one reason initialization and normalization matter.

The right question is not only "is there a gradient?" It is also "is the gradient in a useful range?"