LLM Primer

Derivatives

A derivative measures how a function changes near one input.

For:

f(x)=x2f(x) = x^2

the derivative is:

f′(x)=2xf'(x) = 2x

At x = 3, the derivative is:

f′(3)=6f'(3) = 6

That means a small increase in x near 3 increases f(x) at about six times that small input change.

For a tiny change Delta x, we often use the local approximation:

Δf≈f′(x)Δx\Delta f \approx f'(x)\Delta x

At x = 3, if Delta x = 0.1, then:

Δf≈6(0.1)=0.6\Delta f \approx 6(0.1) = 0.6

This is an approximation, not a promise that the whole curve is a straight line.

The approximation is most trustworthy for small moves near the point where the derivative was measured. As the move gets larger, curvature can make the actual change differ more from the linear estimate.

Derivative as Sensitivity

The derivative is a sensitivity.

If the derivative is large, the output changes quickly. If the derivative is near zero, the output changes slowly.

In training, this helps us decide how strongly to update a parameter.

The sign also matters:

  • positive derivative: increasing the input increases the output locally
  • negative derivative: increasing the input decreases the output locally
  • zero derivative: the output is locally flat in that direction

A Small ML Reading

If L(w) is a loss as a function of one weight w, then:

dLdw\frac{dL}{dw}

tells how the loss changes when that weight changes.

If dL/dw is positive, increasing w increases the loss locally, so a descent step would move w downward. If dL/dw is negative, increasing w decreases the loss locally, so a descent step would move w upward.

MATH-C06-T03-001Exercise: Evaluate a derivative

Let f(x) = x^2, so f'(x) = 2x.

What is f'(4)?

Compute it first, then check your number.

Hint

Substitute x = 4 into 2x.

Solution

The derivative formula is f'(x) = 2x. Substitute the input x = 4:

f'(4) = 2(4) = 8

So near x = 4, the output changes about 8 times as fast as the input.

MATH-C06-T03-002Exercise: Use a local approximation

Let f'(3) = 6. If x increases by 0.1, about how much does f(x) change?

Compute it first, then check your number.

Hint

Use Delta f approx f'(x) Delta x.

Solution

Use the local linear approximation:

Delta f approx f'(x) Delta x

Here the derivative is 6 and the input change is 0.1, so:

Delta f approx 6(0.1) = 0.6

This is an estimate near the point where the derivative was measured.

MATH-C06-T03-003Exercise: Read derivative sign

If dL/dw = -4, does increasing w increase or decrease the loss locally?

Answer it first, then check.

Hint

A negative derivative means output moves opposite the input increase.

Solution

It decreases the loss locally. A small positive change in w gives a negative change in L.

MATH-C06-T03-004Exercise: Avoid a global overclaim

Does f'(3) = 6 mean the function has slope 6 everywhere?

Answer it first, then check.

Hint

A derivative is measured at a point.

Solution

No. f'(3) = 6 describes the local slope near x = 3.

The derivative can change at other inputs. A local slope is useful for nearby changes, but it is not a statement that the whole function has the same slope everywhere.

MATH-C06-T03-005Exercise: Trust local approximations locally

Enter 1 if Delta f approx f'(x) Delta x is generally safer for small moves near x than for large moves far away.

Compute it first, then check your number.

Hint

Ask where the derivative value was measured.

Solution

Enter 1. The derivative gives a local linear approximation near the measured point. Large moves can leave that local region.

Before Moving On

A derivative is a local sensitivity, not a global description of the whole function.