Momentum and Adaptive Optimizers

SGD uses the current gradient directly. Momentum adds memory.

Instead of stepping only from the newest gradient, momentum keeps a velocity. The velocity combines recent gradients, so the update direction changes more smoothly.

Momentum remembers recent update direction, so each step is not based only on the newest gradient.

A simple momentum form is:

velocity = momentum * velocity + gradient
parameter = parameter - learning_rate * velocity

If momentum = 0, the velocity is just the current gradient. If momentum is close to 1, older gradients keep more influence.

Adaptive optimizers, such as Adam and AdamW, go further. They track moving averages of gradients and squared gradients, then adjust step sizes per parameter. AdamW also separates weight decay from the gradient update, which often behaves better in modern neural-network training.

You do not need to memorize Adam's full formula yet. For now, keep the role clear: an optimizer transforms gradients into parameter updates.

Exercise: Momentum velocity

Let momentum = 0.9, previous velocity = 2, and current gradient = 3. Using velocity = momentum * velocity + gradient, what is the new velocity?

Your answer

Compute it first, then check your number.

Exercise: Optimizer role

Enter 1 if an optimizer computes predictions, 2 if it turns gradients into parameter updates, or 3 if it stores the dataset.

Your answer

Compute it first, then check your number.