Stochastic and Mini-Batch Gradient Descent
Full-batch gradient descent uses the whole dataset to compute one update.
Stochastic gradient descent uses one example at a time.
Mini-batch gradient descent uses a small batch of examples.
Why Mini-Batches?
Mini-batches are a practical compromise.
They are cheaper than full-batch updates and less noisy than one-example updates.
This is why mini-batch training is common in deep learning.
Full-batch updates can be expensive because every update waits for the whole dataset. One-example updates are cheap but noisy. Mini-batches sit between those extremes.
One pass through the dataset is called an epoch. If a dataset has 1000
examples and the batch size is 100, one epoch has 10 mini-batches.
Noise Can Help
Mini-batch gradients are estimates of the full gradient.
They contain noise, but that noise can help training move through complicated loss surfaces.
Noise is not automatically good. Too much noise can make training unstable. The useful idea is that a mini-batch gradient can be good enough to improve the model while being much cheaper than a full gradient.
Batch size changes the character of the estimate. Larger batches are usually less noisy but more expensive per update. Smaller batches are cheaper but can jitter more. The right choice is a tradeoff, not a moral rule.
A dataset has 1000 examples and a mini-batch has 100 examples.
How many mini-batches cover one pass through the dataset?
Compute it first, then check your number.
Hint
Solution
1000 / 100 = 10 mini-batches. One epoch covers the dataset once, so we divide
the total example count by the batch size.
Which method uses the whole dataset for one update: full-batch or
mini-batch?
Answer it first, then check.
Hint
The name says whether the whole dataset is used.
Solution
Full-batch gradient descent uses the whole dataset for one update. That makes each update less noisy, but it can be expensive on large datasets.
Is a mini-batch gradient an estimate of the full gradient?
Answer it first, then check.
Hint
Mini-batches use part of the data.
Solution
Yes. A mini-batch gradient estimates the full gradient from a subset of the data.
A dataset has 1200 examples and the batch size is 300.
How many mini-batches cover one epoch?
Compute it first, then check your number.
Hint
Divide dataset size by batch size.
Solution
1200 / 300 = 4 mini-batches. Four batches of 300 examples cover all 1200
examples once, so that completes one epoch.
Enter 1 if increasing batch size usually reduces gradient noise but makes
each update use more examples.
Compute it first, then check your number.
Hint
Compare estimating from 4 examples with estimating from 400 examples.
Solution
Enter 1. A larger mini-batch averages over more examples, so the estimate is
usually less noisy, but each update requires more computation.
Before Moving On
Mini-batches make gradient descent practical for large datasets.