Momentum
Momentum remembers part of the previous update direction.
Instead of using only the current gradient, momentum builds a moving direction over time.
Why It Helps
Momentum can smooth noisy gradients.
It can also help movement continue in directions that consistently reduce loss.
Without momentum, each update may react strongly to the latest mini-batch.
With momentum, the update direction is partly current signal and partly recent history.
Intuition
Imagine walking downhill while carrying some speed from previous steps.
If the slope keeps pointing mostly the same way, momentum helps continue.
If gradients change direction sharply, momentum resists sudden changes.
That resistance is useful when mini-batch gradients jitter around a consistent trend. It can be harmful if the stored direction keeps pushing past a region where the optimizer should slow down or turn.
So momentum is not simply "faster gradient descent." It is a memory term. Memory helps when recent directions contain useful signal, and hurts when recent directions are no longer trustworthy.
Does momentum use information from previous update directions?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
Solution
Yes. Momentum uses previous update direction information. Enter 1. It adds
memory to the update instead of reacting only to the latest gradient.
Does momentum use only the current gradient and forget all previous movement?
Answer it first, then check.
Hint
Momentum remembers part of recent movement.
Solution
No. Momentum combines current gradient information with recent update history. That stored direction can smooth noisy mini-batch updates.
Can momentum help smooth noisy mini-batch gradients?
Answer it first, then check.
Hint
Momentum carries a moving direction over time.
Solution
Yes. Momentum can smooth noisy gradients by carrying information across updates. It averages movement over time instead of letting every mini-batch completely reset the direction.
If gradients point mostly the same way for several steps, does momentum tend to help movement continue in that direction?
Answer it first, then check.
Hint
Momentum carries speed from previous steps.
Solution
Yes. Momentum tends to reinforce directions that remain consistent across updates. Repeated agreement across gradients builds a stronger moving direction.
Enter 1 if momentum can sometimes push past a region where plain gradient
descent would slow down.
Compute it first, then check your number.
Hint
Memory can resist a sudden change in direction.
Solution
Enter 1. Momentum carries recent movement. That can smooth useful motion, but
it can also overshoot when the optimizer should slow down or turn.
Before Moving On
Momentum adds memory to gradient-based updates.