Inspect Gradient Norms and Learning-Rate Sensitivity

Calculate global gradient norm, interpret clipping as magnitude control, and compare learning rates under fixed model, data, seed, optimizer, clipping, and step budget.

The gradient gives a direction in parameter space. The optimizer and learning rate turn that direction and stored optimizer state into an update. A finite gradient does not guarantee that the resulting update is useful.

Calculate One Global Norm

For parameter gradients $g_1,\ldots,g_m$ , the global Euclidean norm is

\lVert g\rVert_2= \sqrt{\sum_{j=1}^{m}\sum_k (g_j)_k^2}.

If this norm exceeds threshold $c$ , norm clipping rescales all gradients by the same factor $c/\lVert g\rVert_2$ . Their joint direction is preserved while their magnitude is bounded.

For example, a norm of 2.5 clipped at 1.0 receives scale $1/2.5=0.4$ .

Clipping must occur after backward and before the optimizer step. It controls gradient magnitude; it does not repair a wrong target, mask, or loss.

Hold the Experiment Fixed when Comparing Rates

The table below changes only learning rate. Every run uses the frozen model, seeds, sampled corpora, batches, AdamW with zero weight decay, clipping at 1.0, and 250 updates.

Learning rate	Final train loss	Final validation loss	Reading
0	2.1898	2.1918	gradients exist, but parameters do not move
0.001	1.1462	1.1528	learning occurs slowly in this budget
0.01	0.5353	0.5276	substantial progress, but not yet at the floor
0.1	0.5431	0.5456	similar progress under this short budget
1.0	1.8647	1.9394	updates are too coarse for the same budget

This does not establish a generally safe Transformer learning rate. Model size, optimizer, schedule, batch, precision, and data all change the result.

Interpret Norms with Other Measurements

Gradient observation	Possible explanations	Next check
absent	disconnected or unregistered parameter	named parameters and graph path
exactly zero	inactive path, symmetry, or genuine local derivative	activation and per-parameter norms
finite and small	near a fit or weak signal	loss trend and update-to-parameter ratio
repeatedly above threshold	unstable scale or threshold chosen too low	unclipped norm distribution and loss
non-finite	invalid forward values or unstable backward computation	first non-finite activation and gradient

A clipping event is a measurement, not automatically a failure. Persistent clipping can, however, conceal an underlying scale problem.

Q1. Calculate a clipped norm

A global gradient norm is 4 and the threshold is 1. What scale multiplies every gradient, and what is the resulting norm?

Your answer

Answer it first, then check.

Hint

Use

c/\lVert g\rVert

Solution

The scale is

1/4=0.25

, giving global norm 1.

Research Reference

Pascanu, Mikolov, and Bengio, On the Difficulty of Training Recurrent Neural Networks

The paper motivates and tests norm clipping for recurrent networks. This chapter uses the mechanism, not its experiments as a universal Transformer threshold.