Why Stacked Linear Maps Need Nonlinearity
Stacking linear maps without activations does not create a truly deeper model family.
Suppose:
h = Ax
y = Bh
Substitute the first equation into the second:
y = B(Ax)
= (BA)x
The two linear maps collapse into one linear map.
Bias terms do not change the main idea. A stack of affine maps without nonlinear activations is still an affine map.
What activations change
With an activation:
h = relu(Ax + a)
y = Bh + b
the middle value h is no longer just a linear transformation of x. The activation bends the computation before the next layer uses it.
That is why activations are not decoration. They are what let depth matter.
If h = 2x and y = 3h, then y equals what multiple of x?
Compute it first, then check your number.
HintSubstitute h
Replace h with 2x.
SolutionWork it out
y = 3h = 3(2x) = 6x.
Enter 1 if activations let stacked layers build nonlinear functions, or 0 if they are only cosmetic.
Compute it first, then check your number.
HintThink about the bend
The activation changes the composition before the next layer.
SolutionWork it out
Without activations, stacked linear maps collapse into one linear map. With activations, the model can build nonlinear functions.