Residual Addition

A residual block computes a change and adds it to the input.

y = x + F(x)

Here, F(x) is the transformation learned by the block. The input x also travels through an identity path.

This changes the job of the block. Instead of learning the whole output from scratch, the block can learn a residual correction.

If x = 10 and F(x) = -2, then:

y = 10 + (-2) = 8

If the useful change is small, residual form lets the block make a small adjustment while preserving much of the input.