Forward Values
Autodiff needs the values computed during the forward pass.
Consider:
L = z^2
The local derivative is:
dL/dz = 2z
To evaluate that derivative, we need the forward value of z.
If z = 7, then:
dL/dz = 14
This is why autodiff systems store or recompute forward values. The derivative rule alone is not enough; it must be evaluated at the actual values from the forward computation.
Values and operations
A useful record contains both:
- which operation produced a value;
- what the value was during the forward pass.
The operation gives the derivative rule. The value lets the rule become a number.
If L = z^2 and the forward value is z = 5, what is dL/dz?
Compute it first, then check your number.
HintUse the derivative rule
The derivative of z^2 is 2z.
SolutionWork it out
dL/dz = 2z = 2 x 5 = 10.
Enter 1 if the forward value of z is needed to evaluate 2z, or 0 if the symbol alone is enough for a numerical gradient.
Compute it first, then check your number.
HintNumerical value
To get a number from 2z, you need z.
SolutionWork it out
The expression 2z becomes a number only after substituting the forward
value of z.