LLM Primer

Partial Derivatives

Many ML functions depend on more than one variable.

For example:

f(x,y)=x2+3yf(x, y) = x^2 + 3y

A partial derivative changes one variable while holding the others fixed.

With respect to x:

∂f∂x=2x\frac{\partial f}{\partial x} = 2x

With respect to y:

∂f∂y=3\frac{\partial f}{\partial y} = 3

At the point (x, y) = (2, 5), the partial derivative with respect to x is:

∂f∂x(2,5)=2(2)=4\frac{\partial f}{\partial x}(2, 5) = 2(2) = 4

The partial derivative with respect to y is still 3, because the y term is linear.

Why Hold Others Fixed?

Holding other variables fixed lets us isolate one source of change.

In a model with many parameters, we want to know how the loss responds to each parameter. A partial derivative gives one piece of that answer.

This does not mean the other variables never change in real training. It means we measure one local sensitivity at a time, then collect those sensitivities together.

That collection is important. A single partial derivative is only one coordinate of the local change story. The gradient puts all those one-coordinate sensitivities into one vector.

Notation

The symbol partial reminds us that the function has multiple inputs.

∂L∂wi\frac{\partial L}{\partial w_i}

means: how does the loss change when parameter w_i changes, while other parameters are held fixed?

A Slightly Richer Example

Let:

g(x,y)=xy+y2g(x, y) = xy + y^2

With respect to x, treat y as a constant:

∂g∂x=y\frac{\partial g}{\partial x} = y

With respect to y, xy contributes x and y^2 contributes 2y:

∂g∂y=x+2y\frac{\partial g}{\partial y} = x + 2y
MATH-C06-T04-001Exercise: Evaluate a partial derivative

Let f(x, y) = x^2 + 3y.

What is partial f / partial y?

Compute it first, then check your number.

Hint

Treat x as fixed and look only at the y term.

Solution

When taking partial f / partial y, treat x as fixed.

The term 3y changes with y at rate 3. The term x^2 does not change with y, so it contributes 0.

Therefore:

partial f / partial y = 3
MATH-C06-T04-002Exercise: Evaluate at a point

For f(x, y) = x^2 + 3y, what is partial f / partial x at (x, y) = (2, 5)?

Compute it first, then check your number.

Hint

First compute partial f / partial x = 2x, then substitute x = 2.

Solution

First differentiate with respect to x while holding y fixed:

partial f / partial x = 2x

Then evaluate at the point (2, 5). Only the x value matters for this partial derivative:

2(2) = 4
MATH-C06-T04-003Exercise: Hold one variable fixed

For g(x, y) = xy + y^2, what is partial g / partial x?

Answer it first, then check.

Hint

When differentiating with respect to x, treat y as a constant.

Solution

When differentiating with respect to x, treat y as a constant.

The term xy changes with x at rate y. The term y^2 has no x in it, so it contributes 0.

So:

partial g / partial x = y
MATH-C06-T04-004Exercise: Avoid a common mistake

When computing partial f / partial x, do we allow y to change at the same time?

Answer it first, then check.

Hint

The point of a partial derivative is to isolate one input.

Solution

No. A partial derivative with respect to x measures change in the x direction while holding the other variables fixed.

MATH-C06-T04-005Exercise: One coordinate of sensitivity

Enter 1 if one partial derivative is one component of the gradient for a scalar-output function.

Compute it first, then check your number.

Hint

Think of each partial derivative as one coordinate in the gradient vector.

Solution

Enter 1. For a scalar-output function, the gradient is the vector of partial derivatives.

Before Moving On

Partial derivatives let us inspect one input direction at a time.