Gradient Descent

Gradient descent updates parameters in the direction that lowers the loss.

The update rule is:

w_{new} = w - \eta \nabla L(w)

Here, eta is the learning rate.

Gradient descent uses local slope to take downhill steps.

In one dimension, the gradient is the slope. In many dimensions, the gradient is a vector of partial derivatives.

Each coordinate says how the loss changes if that parameter moves a little.

Why the Minus Sign?

The gradient points toward steepest increase.

To reduce loss, we move in the opposite direction.

If the gradient is positive, increasing the parameter increases loss, so gradient descent decreases the parameter.

If the gradient is negative, increasing the parameter decreases loss, so gradient descent increases the parameter.

Small Example

If the gradient is 4 and the learning rate is 0.1, the update step is:

0.1 \cdot 4 = 0.4

So the parameter moves by -0.4.

If w = 3, the new value is:

3 - 0.4 = 2.6

Local Information

Gradient descent uses local information. It does not know the whole landscape.

That is why learning rate and loss shape matter. A step can be useful near one point and too large near another.

The gradient answers a narrow question: if we move a tiny amount from here, which direction changes the loss fastest? It does not promise that a finite step will improve the loss. The update is a local suggestion plus a chosen step size.

MATH-C09-T04-001Exercise: Compute a gradient step

Let w = 3, gradient = 4, and learning rate = 0.1.

What is the new w after gradient descent?

Check w

Compute it first, then check your number.

Hint

Use w - learning_rate * gradient.

Solution

3 - 0.1 * 4 = 3 - 0.4 = 2.6. The gradient is positive, so gradient descent moves the parameter downward by 0.4.

MATH-C09-T04-002Exercise: Read the minus sign

If the gradient is positive and the learning rate is positive, does gradient descent increase or decrease the parameter?

Check

Answer it first, then check.

Hint

Use w_new = w - eta * gradient.

Solution

It decreases the parameter because a positive quantity is subtracted from w. This is the minus-sign idea: move opposite the direction of increasing loss.

MATH-C09-T04-003Exercise: Negative gradient

Let w = 2, gradient = -3, and learning rate = 0.1.

What is the new w after gradient descent?

Check w

Compute it first, then check your number.

Hint

Compute 2 - 0.1 * (-3).

Solution

2 - 0.1 * (-3) = 2 + 0.3 = 2.3. A negative gradient means increasing the parameter locally reduces the loss, so the update moves upward.

MATH-C09-T04-004Exercise: Local information

Does gradient descent use local slope information rather than knowing the entire loss landscape at once?

Check yes or no

Answer it first, then check.

Hint

The gradient is computed at the current parameter value.

Solution

Yes. Gradient descent uses the local gradient at the current point. It does not see the whole loss landscape before choosing a finite step.

MATH-C09-T04-005Exercise: Local suggestion

Enter 1 if the gradient gives local direction information, while the learning rate decides how far to trust that direction.

Your answer

Compute it first, then check your number.

Hint

Separate the direction from the distance moved.

Solution

Enter 1. The gradient gives a local direction. The learning rate turns that direction into a finite update.

Before Moving On

Gradient descent turns gradients into parameter updates.