Optimizers

An optimizer updates parameters.

The usual PyTorch training step separates four actions:

optimizer.zero_grad()
pred = model(x)
loss = loss_fn(pred, y)
loss.backward()
optimizer.step()

Read those lines as the training loop you already know:

1. clear old gradients
2. run the forward pass
3. compute the loss
4. compute gradients
5. update parameters

zero_grad() is easy to overlook. PyTorch accumulates gradients by default. That is useful in some cases, but in a normal mini-batch loop each step should start with a clean gradient buffer.

step() is the line that changes parameters. With stochastic gradient descent, the simple update is still:

new parameter = old parameter - learning_rate * gradient

Other optimizers add memory, scaling, or adaptive behavior, but they still use gradients to decide an update.

PyTorch makes the update line short. It does not change what an update means.