Exercises
Use these exercises to check the chapter ideas. Work by hand first.
An input changes from x = 2 to x = 6. The output changes from y = 1 to
y = 13.
What is the average slope?
Compute it first, then check your number.
Let f(x) = x^2, so f'(x) = 2x.
What is f'(5)?
Compute it first, then check your number.
If f'(5) = 10 and x increases by 0.2, about how much does f(x) change?
Compute it first, then check your number.
Let g(x, y) = xy + y^2.
What is partial g / partial x?
Answer it first, then check.
Let f(x, y) = x^2 + y^2.
What is grad f(1, 3)?
Compute it first, then check your number.
If grad L = [2, 6], what direction does gradient descent use before scaling by
the learning rate?
Compute it first, then check your number.
Let grad f = [2, -1] and u = [0, 1].
Compute grad f . u.
Compute it first, then check your number.
A function has 4 outputs and 3 inputs.
What is the shape of its Jacobian?
Answer it first, then check.
If dy/dg = 7 and dg/dx = 3, what is dy/dx?
Compute it first, then check your number.
If one value receives gradient contributions 5 and 8 from two later paths,
what total gradient does it receive?
Compute it first, then check your number.
If f'(x) = 4, a small change Delta x = 0.05 gives the local estimate
Delta f approx 0.2.
Enter 1 if this estimate is a local approximation, not a guarantee for large
moves far from x.
Compute it first, then check your number.
Hint
Ask where the derivative was measured and how far the move goes.
Solution
Enter 1. The estimate uses the derivative at the current point. It is most
reliable for small nearby moves, not arbitrary large moves.
If grad L = [-3, 4], what direction does plain gradient descent use before
scaling by the learning rate?
Compute it first, then check your number.
Hint
Plain gradient descent moves opposite the uphill loss direction.
Solution
Plain gradient descent moves opposite the gradient. Negate each component:
This is the direction before multiplying by the learning rate.
A vector-output function has Jacobian shape 2x3.
Enter 1 if it maps a small 3-dimensional input change to an approximate
2-dimensional output change.
Compute it first, then check your number.
Hint
Use Delta f approx J Delta x.
Solution
Enter 1. A 2x3 Jacobian multiplies a 3-dimensional input-change vector and
produces an approximate 2-dimensional output change.
An upstream gradient is 6. The local derivative on one path is -2.
What gradient passes to the earlier value along that path?
Compute it first, then check your number.
Hint
Use 6 * (-2).
Solution
Backprop multiplies the upstream gradient by the local derivative along the path:
The negative sign means the local operation flips the direction of the sensitivity.
Next
Use the hints only after you have tried the exercises. Use the solutions after you can explain where you got stuck.