LLM Primer

Gradients

A gradient collects partial derivatives into a vector.

For a function:

f(x,y)=x2+y2f(x, y) = x^2 + y^2

the gradient is:

∇f(x,y)=[2x,2y]\nabla f(x, y) = [2x, 2y]

At (3, 4), this becomes:

∇f(3,4)=[6,8]\nabla f(3, 4) = [6, 8]
gradientsteepest increase
The gradient points in the direction of steepest increase.

Meaning

The gradient points in the direction where the function increases fastest.

The negative gradient points in the direction where the function decreases fastest.

This is why gradient descent uses:

wnew=w−η∇L(w)w_{new} = w - \eta \nabla L(w)

The symbol eta is the learning rate.

The minus sign matters. The gradient points uphill for the loss. Gradient descent moves in the opposite direction because training tries to reduce the loss.

The gradient is not automatically the update. The optimizer chooses how to use it. Plain gradient descent uses the negative gradient scaled by a learning rate, but later methods may rescale, average, clip, or adapt that signal.

Gradient as a Vector

Since the gradient is a vector, earlier vector ideas apply:

  • direction matters
  • length matters
  • scaling changes the step size
  • dot products can measure directional change

The length of the gradient tells how sensitive the function is locally. A long gradient means a small movement can change the function a lot. A zero gradient means the first-order local signal is flat.

ML Reading

If a loss depends on many weights, the gradient has one component per weight.

Each component answers a local question:

If this weight changes slightly, while the others are held fixed for this measurement, how does the loss change?

MATH-C06-T05-001Exercise: Compute a gradient

Let f(x, y) = x^2 + y^2.

What is grad f(3, 4)?

Compute it first, then check your number.

Hint

Use [2x, 2y].

Solution

The gradient formula is [2x, 2y]. Substitute x = 3 and y = 4:

grad f(3, 4)
= [2(3), 2(4)]
= [6, 8]

The two entries are the local sensitivities in the x and y directions.

MATH-C06-T05-002Exercise: Choose the descent direction

If grad L(w) = [6, 8], which direction does gradient descent move before scaling by the learning rate?

Compute it first, then check your number.

Hint

Gradient descent uses the negative gradient.

Solution

The gradient points uphill for the loss. Plain gradient descent moves in the opposite direction before applying the learning rate:

-[6, 8] = [-6, -8]

So the unscaled descent direction is [-6, -8].

MATH-C06-T05-003Exercise: Read the gradient components

If grad L = [2, -5], which weight has the larger local sensitivity by magnitude: the first or second?

Answer it first, then check.

Hint

Compare absolute values, not signs.

Solution

The second component has magnitude 5, while the first has magnitude 2, so the second weight has the larger local sensitivity.

MATH-C06-T05-004Exercise: Interpret a zero gradient

If grad f = [0, 0], does the first-order local signal point uphill in any direction?

Answer it first, then check.

Hint

A zero vector has no direction.

Solution

No. A zero gradient means the first-order local slope is flat in every direction, though later optimization chapters will discuss what that can mean.

MATH-C06-T05-005Exercise: Gradient versus update

Enter 1 if the gradient is the local uphill sensitivity signal, while a descent update usually moves in the negative-gradient direction.

Compute it first, then check your number.

Hint

Ask whether training wants to increase or decrease the loss.

Solution

Enter 1. The gradient points toward steepest local increase. To reduce loss, plain gradient descent moves in the opposite direction.

Before Moving On

A gradient is the vector form of local sensitivity.