Gradient Accumulation
A value can influence the loss through more than one path.
When that happens, its gradient contributions add.
Consider:
u = 2x
v = 3x
L = u + v
The loss depends on x through u and through v.
The derivatives are:
dL/du = 1
dL/dv = 1
du/dx = 2
dv/dx = 3
So:
dL/dx = 1 x 2 + 1 x 3 = 5
This is gradient accumulation.
Why accumulation matters
In neural networks, a parameter may affect many examples, many outputs, or many paths in a graph. The final gradient is the sum of all contributions that reach that parameter.
Exercise: Add two gradient paths
A value has two gradient contributions: 7 and -2. What is the accumulated gradient?
Compute it first, then check your number.
HintSum contributions
Contributions to the same value add.
SolutionWork it out
7 + (-2) = 5.
Exercise: Two paths from x
Let u = 2x, v = 5x, and L = u + v. What is dL/dx?
Compute it first, then check your number.
HintEach path contributes
dL/du = 1 and dL/dv = 1.
SolutionWork it out
dL/dx = 1 x 2 + 1 x 5 = 7.