LLM Primer

Projection Shortcut

Residual addition requires compatible shapes.

If x and F(x) have different shapes, they cannot be added directly. A projection shortcut transforms x into a compatible shape:

y = P(x) + F(x)
project shortcut when shapes differxF(x)P(x)+y
A projection shortcut changes the skip path so addition is shape-compatible.

The projection may change channel count, width, or another dimension. It is still a shortcut, but not a pure identity path.

Projection shortcuts are useful when a network changes representation size between stages. They preserve the shortcut idea while letting the model move to a new shape.

DL-C15-T02-001Exercise: Shape compatibility

Enter 1 if two vectors must have compatible shapes to be added elementwise, or 2 if shapes never matter.

Compute it first, then check your number.

DL-C15-T02-002Exercise: Projected residual

Let P(x) = 4 and F(x) = -1. What is y = P(x) + F(x)?

Compute it first, then check your number.