Introduction

Optimization is the process of searching for parameters that reduce a loss.

In machine learning, training is optimization:

measure loss
compute direction
update parameters
repeat

The Main Idea

A model has parameters. A loss function measures how poorly those parameters perform on data.

Optimization changes the parameters to reduce the loss.

The parameters may be millions or billions of numbers. The loss is usually one number. Training asks: how should we change many parameters so this one number goes down?

That is why gradients matter. A gradient gives local information about how the loss changes when parameters move a little.

What Optimization Is Not

Optimization is not the same as understanding, generalization, or intelligence. It is the procedure that adjusts parameters according to an objective.

If the objective is badly chosen, optimization can faithfully improve the wrong thing.

What This Chapter Covers

This chapter introduces objectives, loss functions, minima and maxima, gradient descent, stochastic and mini-batch gradient descent, learning rates, momentum, adaptive methods, convexity, regularization, and numerical stability.

Read the chapter with one question in mind:

What number is being reduced, and what update rule is used to reduce it?

Before Moving On

You should be comfortable with gradients, numerical stability, and the idea that a model output can be compared with a target using a loss.