Gradient Checking for Parameters
Gradient checking compares a backpropagation gradient with a finite-difference estimate.
For a parameter w, a centered finite-difference check is:
[L(w + epsilon) - L(w - epsilon)] / (2 epsilon)
This is slow for large models, but useful for small checks.
Tiny example
Let:
L(w) = w^2
w = 3
epsilon = 1
Then:
[L(4) - L(2)] / 2
= (16 - 4) / 2
= 6
The true derivative is:
dL/dw = 2w = 6
The check agrees.
Let L(w) = w^2, w = 4, and epsilon = 1. Compute [L(5) - L(3)] / 2.
Compute it first, then check your number.
HintEvaluate the two losses
Square 5 and 3.
SolutionWork it out
L(5) = 25 and L(3) = 9, so (25 - 9) / 2 = 8.
Enter 1 if gradient checking is mainly a debugging check for small cases, or 0 if it replaces backpropagation in normal training.
Compute it first, then check your number.
HintCost
Finite differences require rerunning the loss for parameter nudges.
SolutionWork it out
Gradient checking is useful for small debugging cases. It is too expensive to replace backpropagation in normal neural-network training.