Exercises
These exercises check the numerical stability ideas from the chapter.
For logits [3, 10, 4], what value should be subtracted for stable softmax?
Compute it first, then check your number.
For logits [3, 10], after subtracting the maximum, what is the first shifted logit?
Compute it first, then check your number.
Let denominator be 0.004 and epsilon be 0.001. What denominator is used after adding epsilon?
Compute it first, then check your number.
Enter 1 if exp(1000) is an overflow warning, or 2 if it is always safe in ordinary floating-point code.
Compute it first, then check your number.
Enter 1 if NaN should trigger debugging, or 2 if it is a normal finite training value.
Compute it first, then check your number.
Enter 1 if dtype choice affects memory and numerical behavior, or 2 if dtype is only a display setting.
Compute it first, then check your number.
Enter 1 if mixed precision can use lower precision for speed while preserving stability-sensitive work, or 2 if it means ignoring stability.
Compute it first, then check your number.