LLM Primer

Jacobians and Hessians

Gradients handle scalar outputs.

Some functions have vector outputs. For example:

f(x,y)=[x+y,xy]f(x, y) = [x + y, xy]

The Jacobian collects partial derivatives for each output with respect to each input.

If a function has m outputs and n inputs, its Jacobian has shape m x n. Rows correspond to outputs. Columns correspond to inputs.

Jacobian

For the function above:

J=[∂f1/∂x∂f1/∂y∂f2/∂x∂f2/∂y]=[11yx]J = \begin{bmatrix} \partial f_1 / \partial x & \partial f_1 / \partial y \\ \partial f_2 / \partial x & \partial f_2 / \partial y \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ y & x \end{bmatrix}

The Jacobian tells how a small input change affects all outputs.

In local approximation form:

Δf≈JΔx\Delta f \approx J\Delta x

This is the vector-output version of the one-variable approximation Delta f approx f'(x) Delta x.

At (x, y) = (3, 2), this becomes:

J(3,2)=[1123]J(3, 2) = \begin{bmatrix} 1 & 1 \\ 2 & 3 \end{bmatrix}

The entry in row 2, column 1 asks how output f_2 changes when input x changes.

Hessian

The Hessian collects second derivatives of a scalar function.

It tells how the gradient changes.

For this course, keep the working meaning:

  • gradient: local slope direction
  • Hessian: local curvature information

For:

h(x,y)=x2+y2h(x, y) = x^2 + y^2

the gradient is [2x, 2y]. The Hessian is:

[2002]\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}

The diagonal entries say both directions curve upward at rate 2. The zero off-diagonal entries say there is no mixed xy curvature in this simple function.

ML Reading

Jacobians appear when outputs are vectors. Hessians appear in curvature, second-order optimization, and local analysis of loss surfaces.

Most deep learning training does not build full Hessians for large models. They are too large. But the idea matters: curvature tells how the gradient itself is changing.

MATH-C06-T07-001Exercise: Read a Jacobian entry

For f(x, y) = [x + y, xy], what is partial f_2 / partial x?

Compute it first, then check your number.

Hint

f_2 = xy. Differentiate with respect to x, then use y = 2.

Solution

The second output is f_2 = xy. When differentiating with respect to x, treat y as fixed, so:

partial(xy) / partial x = y

At the given point, y = 2, so the Jacobian entry is 2.

MATH-C06-T07-002Exercise: Read Jacobian shape

A function has 3 outputs and 2 inputs.

What is the shape of its Jacobian?

Answer it first, then check.

Hint

Rows correspond to outputs; columns correspond to inputs.

Solution

The Jacobian has one row per output and one column per input, so its shape is 3x2.

MATH-C06-T07-003Exercise: Evaluate a Jacobian entry

For f(x, y) = [x + y, xy], what is partial f_2 / partial y at x = 3?

Compute it first, then check your number.

Hint

f_2 = xy. Differentiate with respect to y.

Solution

The second output is f_2 = xy. When differentiating with respect to y, treat x as fixed, so:

partial(xy) / partial y = x

At the given point, x = 3, so the entry is 3.

MATH-C06-T07-004Exercise: Read Hessian meaning

Does a Hessian describe how the gradient changes?

Answer it first, then check.

Hint

The Hessian collects second derivatives.

Solution

Yes. The Hessian describes local curvature, which means it describes how the gradient changes near a point.

MATH-C06-T07-005Exercise: Jacobian as local map

A function has vector output and local approximation Delta f approx J Delta x.

Enter 1 if the Jacobian acts like the local linear map from input changes to output changes.

Compute it first, then check your number.

Hint

Compare with Delta f approx f'(x) Delta x for one-variable functions.

Solution

Enter 1. For vector-output functions, the Jacobian is the local linear map that approximates how small input changes affect the outputs.

Before Moving On

Jacobians track vector-output sensitivity. Hessians track curvature.