LLM Primer

Learning Rate

The learning rate controls step size.

smallusefultoo largelearning rate controls step size
A learning rate that is too small is slow; one that is too large can overshoot.

If the learning rate is too small, training can be slow.

If it is too large, training can overshoot useful regions or diverge.

The learning rate does not choose the direction. The gradient chooses the local direction. The learning rate decides how far to move in that direction.

This is why the same gradient can be helpful or harmful depending on scale. A small step may lower the loss. A very large step may jump across the useful region and land where the loss is worse.

Update Rule

In:

wnew=w−η∇L(w)w_{new} = w - \eta \nabla L(w)

eta scales the update.

For the same gradient, a larger eta produces a larger step.

If the gradient is 6, then:

  • with eta = 0.1, the step size is 0.6
  • with eta = 0.5, the step size is 3

ML Reading

Learning rate is often one of the most important training choices.

Schedules can change the learning rate over time. A training run may start with larger steps and later use smaller steps.

This is not arbitrary. Early training may need broad movement. Later training may need smaller adjustments near a useful region.

When training diverges, the learning rate is one of the first things to inspect. It is not the only possible cause, but it directly controls update size.

MATH-C09-T06-001Exercise: Scale a step

If the gradient is 6 and the learning rate is 0.5, what is the step size before applying the minus sign?

Compute it first, then check your number.

Hint
Multiply learning rate by gradient size.
Solution

0.5 * 6 = 3. The learning rate scales the gradient before the minus sign chooses the descent direction.

MATH-C09-T06-002Exercise: Compare two rates

The gradient is 6. Which learning rate gives a larger step: 0.1 or 0.5?

Compute it first, then check your number.

Hint

The learning rate multiplies the gradient.

Solution

0.5 gives the larger step because it multiplies the same gradient by a larger number.

MATH-C09-T06-003Exercise: Direction or distance

Does the learning rate mainly control update direction or update distance?

Answer it first, then check.

Hint

The gradient gives the local direction.

Solution

The learning rate mainly controls update distance, or step size. The gradient supplies the local direction; the learning rate decides how far to move.

MATH-C09-T06-004Exercise: Schedule intuition

Can a learning-rate schedule use larger steps early and smaller steps later?

Answer it first, then check.

Hint

The ML reading describes this pattern.

Solution

Yes. Many schedules start with larger steps and later reduce the learning rate.

MATH-C09-T06-005Exercise: Overshooting

Enter 1 if a learning rate can be too large even when the gradient direction is correct locally.

Compute it first, then check your number.

Hint

Imagine stepping downhill so far that you jump past the valley.

Solution

Enter 1. The gradient gives a local direction. If the learning rate is too large, the finite step can overshoot and make loss worse.

Before Moving On

Learning rate turns gradient direction into an actual step.