Conclusion

This chapter introduced optimization as training by repeated adjustment.

The main idea is simple, but the details matter: choose a number to reduce, estimate a useful direction, choose a step size, update parameters, and check what happened.

You learned that:

• objectives define what training improves, for better or worse
• loss functions turn model behavior into a number
• minima and maxima describe search targets
• gradient descent moves opposite the gradient when minimizing
• mini-batches estimate gradients from part of the data
• learning rate controls update distance
• momentum adds memory to noisy updates
• adaptive methods scale updates using recent gradient statistics
• convexity is a useful special case, not the usual deep learning landscape
• regularization adds preferences to the objective, not guarantees
• numerical stability is part of real training

What Changed In Your Reading

Before this chapter, "training" may have sounded like one black-box action.

After this chapter, you can read a training loop as a sequence of choices:

• What loss is being reduced?
• What data estimates the gradient?
• How large is the step?
• Is the optimizer adding memory or adaptive scaling?
• Are the numbers stable?
• Does validation support the training improvement?

What Comes Next

Statistics comes next.

Optimization trains a model. Statistics helps us judge what the trained model means, how well it generalizes, and how much uncertainty remains.

Keep This Question Nearby

When training changes a model, ask:

What objective is this update trying to reduce?

Then ask the second question:

Is reducing that objective actually improving the behavior we care about?

That pair of questions keeps optimization in its proper place. It is powerful, but it is not a substitute for choosing the right objective, checking scale, and testing behavior on data the model did not fit.