Conclusion

This chapter explained why activations are central to neural networks.

A linear layer computes a score or pre-activation. An activation transforms that value. ReLU clips negative inputs and passes positive inputs through. Sigmoid and tanh squash values into bounded ranges. Saturation explains why flat regions can weaken gradients. Hidden units can be read as detectors whose responses are combined by later layers.

Most importantly, activations make depth meaningful. Without nonlinear activations, stacked linear maps collapse into another linear map.

The next chapter uses this idea to build multilayer perceptrons.