Exercises

Let yhat = 8 and y = 5.

Compute (yhat - y)^2.

Let w = 10, gradient = 6, and learning rate = 0.5.

What is the new w?

A dataset has 1200 examples and each mini-batch has 300 examples.

How many mini-batches cover one pass through the dataset?

If data loss is 12 and penalty is 3, what is their sum?

If learning rate is 0.1 and gradient is 20, what is the step size before the minus sign?

If the training loss improves, does that automatically prove the reported evaluation metric improved?

For a convex function, is every local minimum also a global minimum?

If data loss = 8, penalty = 4, and lambda = 0.25, what is data loss + lambda * penalty?

Does momentum use information from previous update directions?

Is NaN loss by itself a complete diagnosis of the cause?

Enter 1 if a loss is the number the optimizer can reduce, but it may still be only a proxy for the behavior we care about.

Enter 1 if a gradient can point in a useful local direction while a learning rate that is too large still makes the update harmful.

Enter 1 if larger mini-batches usually reduce gradient noise but cost more per update.

Enter 1 if the guarantee "every local minimum is global" requires convexity and should not be assumed for general deep neural network losses.

Enter 1 if regularization changes the training preference but held-out data is still needed to check whether generalization improved.