Summary and Revision Notes

Key ideas

• A residual block computes y = x + F(x).
• A projection shortcut changes the skip path when shapes differ.
• A skip connection routes information around transformations.
• Concatenation joins features instead of adding them.
• Gates learn how much signal to pass, keep, or mix.
• Identity paths help gradients move through deep networks.
• Residual blocks made very deep training easier by making small corrections and identity behavior easier.

Common formulas

residual_output = x + F(x)
projected_residual_output = P(x) + F(x)
gated_output = gate * candidate + (1 - gate) * old
dy/dx for y = x + F(x): 1 + dF/dx

Common mistakes

• Forgetting that residual addition requires compatible shapes.
• Treating concatenation as the same operation as addition.
• Thinking skip connections remove learning.
• Assuming gates are only on/off switches.
• Missing the optimization role of identity paths.

Before moving on

You should be able to compute a residual sum, explain projection shortcuts, compare addition with concatenation, compute a simple gate, and describe why identity paths help gradients.