LLM Primer

Directional Derivatives

A directional derivative asks:

How fast does a function change if we move in this direction?

If u is a unit direction, then:

Duf(x)=∇f(x)⋅uD_u f(x) = \nabla f(x) \cdot u

This is a dot product between the gradient and a direction.

The direction u should be a unit vector. Otherwise the dot product mixes two things: direction and step length.

Connection to Geometry

If the direction points with the gradient, the function increases quickly.

If the direction is orthogonal to the gradient, the directional derivative is zero.

If the direction points against the gradient, the function decreases.

This is the same dot-product idea from vectors. The gradient is the steepest uphill direction. The directional derivative asks how much of that uphill direction lies inside the direction you chose.

So a directional derivative is not a new kind of derivative to memorize. It is the gradient viewed from one chosen direction.

Why ML Cares

A parameter update is a direction in parameter space. The directional derivative tells whether that update should increase or decrease the loss locally.

If the directional derivative is negative, moving that way should reduce the loss locally. If it is positive, moving that way should increase the loss locally.

MATH-C06-T06-001Exercise: Compute directional change

Let grad f = [3, 4] and u = [1, 0].

Compute grad f . u.

Compute it first, then check your number.

Hint

Use the dot product.

Solution

Use the dot product between the gradient and the unit direction:

[3, 4] . [1, 0]
= 3(1) + 4(0)
= 3

So movement in the horizontal direction has local rate of change 3.

MATH-C06-T06-002Exercise: Check an orthogonal direction

Let grad f = [3, 4] and u = [4/5, -3/5].

What is grad f . u?

Compute it first, then check your number.

Hint

Compute 3(4/5) + 4(-3/5).

Solution

Compute the dot product:

3(4/5) + 4(-3/5)
= 12/5 - 12/5
= 0

The positive and negative contributions cancel. This chosen direction is orthogonal to the gradient, so the function is locally flat along it.

MATH-C06-T06-003Exercise: Read a descent direction

If a directional derivative is -2, does moving in that direction increase or decrease the function locally?

Answer it first, then check.

Hint

A negative directional derivative means the output goes down for a small step in that direction.

Solution

A negative directional derivative means that a small step in that direction makes the function go down.

Here the value is -2, so moving in that direction decreases the function locally.

MATH-C06-T06-004Exercise: Remember the unit direction

Why do we usually require u to be a unit vector?

Enter direction if it is to measure direction without mixing in step length.

Answer it first, then check.

Hint

If u is longer, the dot product becomes larger partly because of length.

Solution

Using a unit vector lets the directional derivative measure change per unit movement in that direction, without mixing in the length of u.

MATH-C06-T06-005Exercise: View from one direction

Enter 1 if a directional derivative can be computed as the dot product of the gradient with a unit direction.

Compute it first, then check your number.

Hint

The formula is D_u f(x) = grad f(x) . u when u is unit length.

Solution

Enter 1. A directional derivative measures how much of the gradient lies in the chosen unit direction.

Before Moving On

Directional derivatives connect gradients to dot products and movement.