Gradient Checks
A gradient check compares a backpropagation gradient with a numerical estimate.
The goal is simple: compute the same local slope in two different ways and see whether they agree.
If they do not agree, the model may still run, but the training signal may be wrong.
Finite Difference
For one variable:
approximates the derivative near x.
If this estimate and the backpropagation gradient disagree strongly, something may be wrong.
The centered version uses both sides of the point. It often gives a better
local estimate than looking only at f(x + h) - f(x).
What Can Go Wrong
The step size h must be chosen with care.
If h is too large, the estimate is no longer very local.
If h is too small, rounding error can dominate the subtraction.
So gradient checking has its own numerical tradeoff. A mismatch can reveal a
bug, but it can also come from a poorly chosen h or a nonsmooth point. The
check is most useful when the small example and step size are chosen carefully.
Why Not Always Use This?
Finite differences are useful for checking, but too slow for normal training.
Backpropagation is efficient because it reuses graph structure.
For a model with millions of parameters, checking each parameter by perturbing it separately would be far too expensive for every training step. Gradient checks are best used as debugging tools on small examples.
Let f(x + h) = 9, f(x - h) = 5, and h = 2.
Compute (f(x+h) - f(x-h)) / (2h).
Compute it first, then check your number.
Hint
2h = 4.Solution
(9 - 5) / 4 = 1. The numerator measures the change across the two nearby
points, and the denominator 2h is the distance between those points.
In a gradient check, do we compare a backpropagation gradient with a finite difference estimate?
Answer it first, then check.
Hint
The first sentence names the two quantities.
Solution
Yes. A gradient check compares the backpropagation gradient with a finite difference estimate.
For centered difference with h = 0.01, what is the denominator 2h?
Compute it first, then check your number.
Hint
Compute 2 * 0.01.
Solution
The denominator is 2h = 2 * 0.01 = 0.02. Centered difference compares values
on both sides of the point, so the total separation is twice the step size.
Why are finite-difference gradients not used as the normal training method for large neural networks?
Enter 1 for because they are too slow, or 2 for because they never estimate
derivatives.
Compute it first, then check your number.
Hint
Each parameter would need separate perturbations.
Solution
Finite differences are too slow for normal training in large networks. Enter
1.
Enter 1 if finite-difference gradient checks can be unreliable when h is
too large or too small.
Compute it first, then check your number.
Hint
Large h leaves the local region; tiny h makes subtractive rounding error
matter.
Solution
Enter 1. If h is too large, the estimate is not local. If h is too small,
rounding error can dominate the subtraction.
Before Moving On
Gradient checks are tests for trust, not a replacement for backpropagation.