Data, Batches, and Epochs

Training rarely updates from the whole dataset at once. It usually uses batches.

A batch is a small group of examples used for one training step. An epoch is one pass over the training set. A step is one parameter update.

Those words describe different counts:

examples -> batches -> steps -> epochs

If a dataset has 100 examples and the batch size is 10, one epoch contains 10 steps. If training runs for 3 epochs, the loop performs 30 parameter updates.

Batches make training practical. They also make gradients noisy. One batch may not represent the whole dataset perfectly, so its gradient is an estimate of the direction that should reduce average loss.

That noise is not only a defect. In many settings it helps the optimizer avoid brittle behavior. But it also means one step can look worse even when the longer trend is improving.