Review

Key ideas

• A computation graph records operations and dependencies.
• Forward values are needed to evaluate local derivatives.
• Local derivatives describe one operation.
• Chain rule combines local derivatives.
• Gradient contributions add when paths merge.
• Stop-gradient uses a value in the forward pass but blocks gradient flow.
• Finite differences estimate derivatives by rerunning nearby inputs.
• Reverse mode is the usual fit for many parameters and one scalar loss.
• Autodiff automates derivative bookkeeping, not modeling judgment.

Common formulas

dL/dx = dL/du * du/dx
accumulated gradient = sum of path contributions
finite difference = [f(x + eps) - f(x - eps)] / (2 eps)

Common mistakes

• Thinking autodiff is symbolic algebra.
• Thinking finite differences are the main training method.
• Forgetting to add gradient contributions from multiple paths.
• Forgetting that local derivatives need forward values.
• Treating stop-gradient as changing the forward value.

Before moving on

You should be able to trace a small computation graph, compute a chain-rule derivative, add two gradient contributions, and explain why reverse mode is useful for neural networks.