Solutions

MATH-C09-C-001

MATH-C09-C-002

MATH-C09-C-003

MATH-C09-C-004

MATH-C09-C-005

MATH-C09-C-006

No. The training loss and reported evaluation metric can be different. A model can improve the optimized loss without automatically improving every metric we care about.

MATH-C09-C-007

Yes. For a convex function, every local minimum is also a global minimum.

MATH-C09-C-008

MATH-C09-C-009

Yes. Momentum uses information from previous update directions, so the next update is not based only on the current gradient.

MATH-C09-C-010

No. NaN loss is a symptom. The cause might be overflow, division by zero, unsafe logarithms, too large a learning rate, or a bug in the code.

MATH-C09-C-011

Enter 1.

A loss is the number the optimizer can reduce. It is useful because it gives a trainable signal, but it is still a proxy. If the proxy is badly chosen, the optimizer may improve the number without improving the behavior we wanted.

MATH-C09-C-012

Enter 1.

The gradient gives local direction information. A finite update also needs a step size. If the learning rate is too large, the update can overshoot and make the loss worse even when the local direction was reasonable.

MATH-C09-C-013

Enter 1.

A larger mini-batch averages over more examples, so its gradient estimate is usually less noisy. The cost is that each update must process more examples.

MATH-C09-C-014

Enter 1.

For a convex objective, every local minimum is global. General deep neural network losses are not usually treated as convex, so that guarantee should not be assumed.

MATH-C09-C-015

Enter 1.

Regularization changes the training objective by adding a preference or penalty. Whether that preference improves generalization must be checked on held-out data.