Chapter 9
Optimization
Loss functions, gradient descent, learning rates, momentum, regularization, and training as search.
What this chapter does
Optimization turns learning into repeated measurement and adjustment. This chapter teaches how losses become search signals, how gradients suggest directions, and why step size, batches, momentum, regularization, and stability determine whether training behaves.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Why optimization is the training language of modern models.
- 02Objectives and Loss Functions
Turning a learning goal into a number the model can reduce.
- 03Minima and Maxima
The low and high points that optimization searches around.
- 04Gradient Descent
Using the gradient to choose a direction that lowers loss.
- 05Stochastic and Mini-Batch Gradient Descent
Estimating useful update directions from part of the data.
- 06Learning Rate
The step size that controls how much each update changes.
- 07Momentum
Remembering recent directions to smooth noisy updates.
- 08Adaptive Methods
Optimizers that adjust step sizes from past gradients.
- 09Convexity
A special case where local low points are global low points.
- 10Regularization
Adding a preference for simpler or safer solutions.
- 11Numerical Stability in Training
Keeping optimization from failing because numbers misbehave.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What Chapter 9 accomplished and how it prepares statistics.
Summary and Revision NotesA compact review of losses, updates, and training behavior.
ExercisesChapter-level practice for optimization.
HintsLow-spoiler nudges for the Chapter 9 exercises.
SolutionsExplained solutions for the Chapter 9 exercises.
Before moving on
- Read optimization as repeated measurement, direction, and step size.
- Understand why learning rate, batches, regularization, and stability matter.
- Explain why a lower training loss is useful evidence, not a guarantee of learning.
- Connect gradient descent to training loops.
Where this leads
- Deep Learning
- Reinforcement Learning