Exercises

These exercises check the full training-loop picture.

Exercise: Count steps

A training set has 120 examples and batch size 20. How many steps are in one epoch?

Compute it first, then check your number.

Exercise: Count updates

If one epoch has 6 steps and training runs for 5 epochs, how many updates happen?

Compute it first, then check your number.

Exercise: SGD update

Let w = 1.5, gradient = -2, and learning_rate = 0.1. What is w_next?

Compute it first, then check your number.

Exercise: Loss derivative

Let L = (y_hat - y)^2, y_hat = 9, and y = 5. What is dL/dy_hat?

Compute it first, then check your number.

Exercise: Momentum velocity

Let momentum = 0.5, previous velocity = 6, and current gradient = 2. What is the new velocity?

Compute it first, then check your number.

Exercise: Clipped gradient

A gradient -9 is clipped to maximum magnitude 4. What gradient is used after clipping?

Compute it first, then check your number.

Exercise: Training signal

Enter 1 if lower training loss always proves better generalization, or 2 if validation behavior is still needed.

Compute it first, then check your number.