Vanishing Gradients

A vanishing gradient becomes very small as it travels backward through many steps.

If each step multiplies a signal by 0.5, then after four steps:

0.5 * 0.5 * 0.5 * 0.5 = 0.0625

The training signal reaching early positions can become weak. Then the model struggles to learn that early tokens mattered.

This is not only an RNN problem, but recurrence makes the repeated path easy to see.

Exercise

If a gradient is multiplied by 0.5 twice, what is the result times 4?