Exploding Gradients

An exploding gradient grows too large as it travels backward through repeated steps.

If each step multiplies a signal by 2, then after five steps:

2 * 2 * 2 * 2 * 2 = 32

Large gradients can make updates unstable. Parameters may jump too far, losses may spike, and numerical values may overflow.

Gradient clipping is one practical tool: if the gradient norm is too large, it is scaled down before the optimizer update.

Exercise

If a signal is multiplied by 2 for three steps, what is the result?