Chapter 9

Training Loops and Optimizers

Batches, epochs, forward-loss-backward-update loops, SGD, momentum, Adam-style optimizers, learning rates, clipping, and training metrics.

Subject
Deep Learning
Lessons
7 lessons
Practice
21 checks
Review
3 pages

Introduction

Backpropagation gives gradients. A training loop decides what to do with them.

A loop is not a new model. It is the repeated procedure that turns one model into a better model: choose data, run a forward pass, compute loss, run the backward pass, update parameters, and check whether the change helped.

batchforwardlossbackwardupdateparametersone step repeats many times
A training step measures error, computes gradients, updates parameters, and repeats on another batch.

This chapter keeps the loop small enough to inspect. We will use scalar and NumPy-style examples, not a framework. That makes the roles visible before a later framework chapter compresses them into shorter code.

By the end, you should be able to read a tiny training loop and say what each line is responsible for.

Lessons

Read these in order.

The chapter opening gives the main idea. Move through these lessons next; each page reuses ideas from the pages before it.

  1. 01
    Loop Anatomy

    The repeated pattern of batch, forward pass, loss, backward pass, update, and logging.

  2. 02
    Data, Batches, and Epochs

    How examples become batches, batches become steps, and epochs count passes over data.

  3. 03
    Forward, Loss, Backward, Update

    Separating prediction, error measurement, gradient computation, and parameter change.

  4. 04
    Stochastic Gradient Descent

    SGD as the basic rule that turns batch gradients into parameter updates.

  5. 05
    Momentum and Adaptive Optimizers

    Momentum, Adam, and AdamW as practical ways to transform gradients into updates.

  6. 06
    Learning Rates and Clipping

    Step size, schedules, and gradient clipping as controls for stable training.

  7. 07
    Tracking Metrics and Failures

    Using loss and validation behavior to notice learning, instability, and overfitting.

  1. •
    Conclusion

    What the full training loop adds before scale and normalization.

  2. •
    Review

    A compact review of training steps, SGD, momentum, learning rates, clipping, and metrics.

  3. •
    Exercises

    Chapter-level practice for training loops and optimizer updates.

You are ready when

  • Trace one complete training step.
  • Count batches, steps, epochs, and updates.
  • Compute scalar SGD and momentum updates.
  • Use metrics to recognize stuck, unstable, and overfitting loops.

Where this leads

  • Initialization, Scale, and Normalization
  • Regularization and Generalization
  • PyTorch as Compression

Chapter progress