Stochastic Gradient Descent

Stochastic gradient descent, or SGD, updates parameters using gradients from one batch.

The rule is the same scalar update you have already seen:

parameter = parameter - learning_rate * gradient

The word stochastic means the update uses a sample of the data, not necessarily the whole dataset. With mini-batches, the gradient is noisy but cheaper to compute.

SGD is worth learning carefully because many advanced optimizers are variations on the same idea. They still look at gradients. They still choose a step. They still change parameters.

The learning rate controls how much of the gradient direction is used. Too small, and training may crawl. Too large, and training may jump across useful regions or diverge.

w = 2.0
gradient = -3.0
learning_rate = 0.1

w_next = 2.0 - 0.1 * (-3.0)
       = 2.3

The parameter increased because the gradient was negative.