LLM Primer

Identity Gradient Path

Backpropagation sends gradients backward through the computation graph.

If a signal must pass through many transformations, gradients must pass through those transformations too. Skip paths create shorter routes.

skip paths give gradients a shorter routexblock 1block 2block 3gradients need not pass only through every nonlinear transform
Identity paths help information and gradients move through deeper networks.

For a residual block:

y = x + F(x)

the derivative of y with respect to x includes an identity term:

dy/dx = 1 + dF/dx

That 1 matters. It gives the gradient a direct path through the addition, even if the transformed path is difficult.

This does not solve every training problem, but it makes very deep networks easier to optimize.

DL-C15-T06-001Exercise: Identity derivative

For y = x + F(x), what constant term appears in dy/dx from the identity path?

Compute it first, then check your number.

DL-C15-T06-002Exercise: Gradient route

Enter 1 if skip paths can give gradients shorter routes, or 2 if they remove all gradients.

Compute it first, then check your number.