Exercises
Use these exercises to check the chapter ideas. Work by hand first, then check the result.
If a computed value is too large for the numeric format, is that overflow or underflow?
Enter 1 for overflow, 2 for underflow.
Compute it first, then check your number.
For logits [4, 10, 7], what value should be subtracted before exponentiating?
Compute it first, then check your number.
For values [-2, 3, 1], what is m in stable log-sum-exp?
Compute it first, then check your number.
What is 0.1 * 0.1 * 0.1?
Compute it first, then check your number.
Let f(x+h) = 14, f(x-h) = 8, and h = 3.
Compute (f(x+h) - f(x-h)) / (2h).
Compute it first, then check your number.
Is floating-point arithmetic the same as exact real-number arithmetic?
Answer it first, then check.
If epsilon is chosen too large, can it noticeably change the intended result?
Answer it first, then check.
An input error of 0.01 changes output A by 0.02 and output B by 20.
Which output shows stronger sensitivity: A or B?
Answer it first, then check.
If many local derivative factors are smaller than 1, do gradients tend to
explode or vanish?
Answer it first, then check.
Why are finite-difference gradients not used as the normal training method for large neural networks?
Enter 1 for because they are too slow, or 2 for because they never estimate
derivatives.
Compute it first, then check your number.
Enter 1 if floating-point results are often compared with a tolerance because
rounded computations can differ slightly.
Compute it first, then check your number.
Hint
Think about two mathematically equivalent computations that round intermediate values differently.
Solution
Enter 1. Floating-point arithmetic can introduce small rounding differences,
so many checks ask whether values are close enough.
Enter 1 if stable softmax changes the intermediate exponentials while keeping
the final probabilities the same.
Compute it first, then check your number.
Hint
Subtracting the same maximum from every logit preserves differences.
Solution
Enter 1. Stable softmax shifts logits before exponentiating. This changes
intermediate values, but the common factor cancels, so probabilities are the
same.
Enter 1 if a finite-difference gradient check can fail when h is chosen too
large or too small.
Compute it first, then check your number.
Hint
Large steps leave the local slope region; tiny steps subtract nearly equal numbers.
Solution
Enter 1. If h is too large, the estimate is not very local. If h is too
small, rounding error can dominate the subtraction.
Enter 1 if epsilon should be used for a known numerical danger, not added to
every formula by habit.
Compute it first, then check your number.
Hint
Ask whether the formula risks division by zero, square root of zero, or log of zero.
Solution
Enter 1. Epsilon can prevent failures such as division by zero or log(0),
but it also changes the expression, so it should be used deliberately.
Next
Use the hints only after you have tried the exercises. Use the solutions after you can explain where you got stuck.