LLM Primer

Solutions

MATH-C06-C-001

The output change is 13 - 1 = 12.

The input change is 6 - 2 = 4.

So the average slope is:

12/4=312 / 4 = 3

MATH-C06-C-002

f′(5)=2(5)=10f'(5) = 2(5) = 10

MATH-C06-C-003

Use the local approximation:

Δf≈f′(x)Δx\Delta f \approx f'(x)\Delta x

So:

Δf≈10(0.2)=2\Delta f \approx 10(0.2) = 2

MATH-C06-C-004

For:

g(x,y)=xy+y2g(x, y) = xy + y^2

with respect to x, treat y as constant. The term xy changes at rate y, and y^2 contributes 0.

So:

∂g/∂x=y\partial g / \partial x = y

MATH-C06-C-005

The gradient is:

∇f(x,y)=[2x,2y]\nabla f(x, y) = [2x, 2y]

So:

∇f(1,3)=[2,6]\nabla f(1, 3) = [2, 6]

MATH-C06-C-006

Gradient descent moves in the negative gradient direction.

So:

−[2,6]=[−2,−6]-[2, 6] = [-2, -6]

MATH-C06-C-007

[2,−1]⋅[0,1]=2(0)+(−1)(1)=−1[2, -1] \cdot [0, 1] = 2(0) + (-1)(1) = -1

MATH-C06-C-008

A Jacobian has one row per output and one column per input.

With 4 outputs and 3 inputs, the shape is:

4×34 \times 3

MATH-C06-C-009

dydx=dydgdgdx=7⋅3=21\frac{dy}{dx} = \frac{dy}{dg}\frac{dg}{dx} = 7 \cdot 3 = 21

MATH-C06-C-010

When a value affects the loss through two later paths, the gradient contributions add:

5+8=135 + 8 = 13

MATH-C06-C-011

The estimate:

Δf≈f′(x)Δx\Delta f \approx f'(x)\Delta x

uses the derivative measured at the current point. It is a local linear approximation, so it is safest for small moves near that point.

MATH-C06-C-012

Plain gradient descent uses the negative gradient.

If:

∇L=[−3,4]\nabla L = [-3,4]

then the descent direction before learning-rate scaling is:

−∇L=[3,−4]-\nabla L = [3,-4]

MATH-C06-C-013

A 2x3 Jacobian has two output rows and three input columns.

In the local approximation:

Δf≈JΔx\Delta f \approx J\Delta x

the input change has length 3, and the approximate output change has length 2.

MATH-C06-C-014

Backprop multiplies the upstream gradient by the local derivative:

6(−2)=−126(-2) = -12

So the gradient passed to the earlier value along that path is -12.