Skip Connection
A skip connection sends information around one or more transformations.
Residual addition is one kind of skip connection. The broader idea is the path: a signal can bypass part of the network and meet a later computation.
Skip connections help because deep networks do not have to carry every useful detail through every transformation. Some information can move forward more directly.
They also make optimization easier. If a block is not useful yet, the skip path can still carry signal.
The key question is:
what information can pass around this block?
That question applies to CNNs, transformers, and many other architectures.
Enter 1 if a skip connection bypasses one or more transformations, or 2 if it must pass through every transformation.
Compute it first, then check your number.
Enter 1 if skip paths can help preserve information, or 2 if they only delete information.
Compute it first, then check your number.