Review

Key ideas

• A training step repeats: batch, forward, loss, backward, update.
• A step is one parameter update.
• An epoch is one pass over the training set.
• SGD updates parameters using batch gradients.
• The learning rate scales the update.
• Momentum keeps memory of recent gradient directions.
• Adam and AdamW are adaptive optimizers that transform gradients into practical updates.
• Gradient clipping limits unusually large gradients.
• Training metrics help diagnose whether the loop is learning, stuck, unstable, or overfitting.

Common formulas

w_next = w - learning_rate * dL/dw
steps_per_epoch = number_of_examples / batch_size
velocity = momentum * velocity + gradient

Common mistakes

• Confusing the forward pass with learning.
• Updating parameters before computing gradients.
• Treating one noisy batch loss as the whole training story.
• Assuming lower training loss always means better generalization.
• Ignoring flat, exploding, or non-finite loss curves.

Before moving on

You should be able to trace one training step, compute a scalar SGD update, count steps from batch size, and explain what an optimizer does.