Forward Mode and Reverse Mode
Autodiff has two important modes.
Forward mode carries derivative information forward with the values.
Reverse mode first runs the forward computation, then sends gradients backward from the output.
Neural network training usually has many parameters and one scalar loss. Reverse mode is efficient for this shape of problem because one backward pass can compute gradients for many parameters.
Intuition
Forward mode asks:
If this input changes, how do later values change?
Reverse mode asks:
If the final loss changes, which earlier values contributed how much?
Both are chain rule. They organize the chain rule in different directions.
Neural network training often has many parameters and one scalar loss. Enter 1 for reverse mode as the usual fit, or 0 for forward mode.
Compute it first, then check your number.
HintMany inputs, one output
Think gradients of one loss with respect to many parameters.
SolutionWork it out
Reverse mode starts from the scalar loss and propagates gradients backward to many earlier values, including many parameters.
Enter 1 if reverse mode sends gradients backward from the output, or 0 if it sends them forward from the input.
Compute it first, then check your number.
HintName says reverse
Reverse mode begins after a forward pass has produced the output.
SolutionWork it out
Reverse mode runs the forward pass first, then sends gradients backward from the output.