Summary and Revision Notes
Key Ideas
| Idea | Meaning |
|---|---|
| objective | quantity optimized by training |
| loss | objective usually minimized |
| metric | number used to evaluate behavior, not always the training loss |
| gradient descent | update opposite the gradient |
| mini-batch | small sample used to estimate an update |
| learning rate | scale of the update step |
| momentum | memory of recent update direction |
| adaptive method | optimizer that changes effective step sizes |
| convexity | special case where local minima are global |
| regularization | added preference or penalty |
| stability check | inspection for numerical training failure |
The important habit is to keep the pieces separate. A loss is not the whole task. A gradient is not the whole landscape. A learning rate is not a direction. An optimizer choice is not evidence of generalization.
Formulas to Remember
Squared loss:
Gradient descent:
Regularized objective:
How To Read A Training Loop
When you see training code, identify these pieces:
- the objective or loss being reduced
- the data used for each update
- the optimizer and update rule
- the learning rate and schedule
- any momentum or adaptive scaling
- any regularization or penalty
- stability checks such as loss, gradient norm, and update size
This turns a training loop from a black box into a set of choices.
Common Patterns
| Pattern | Meaning |
|---|---|
| mini-batch training | cheaper noisy estimate of the full gradient |
| learning-rate schedule | larger or smaller steps at different training stages |
| momentum | smooth recent directions |
| Adam/RMSProp/Adagrad | adapt effective step sizes from gradient history |
| weight decay or penalties | prefer some parameter patterns over others |
| gradient norm checks | detect unstable updates |
Common Traps
- Optimizing a loss that does not match the desired behavior.
- Moving with the gradient when minimizing.
- Choosing a learning rate without checking loss behavior.
- Treating adaptive optimizers as a substitute for understanding scale.
- Assuming convex guarantees apply to most deep networks.
- Treating regularization as a guarantee of generalization.
- Ignoring validation and generalization.
- Treating numerical failures as only modeling failures.
- Treating
NaNas a diagnosis instead of a symptom. - Forgetting that a useful local direction can become a bad finite step.
- Assuming an adaptive optimizer removes the need to reason about scale.
Mental Model
Optimization is a loop: measure, compute direction, step, and check.