Introduction

Linear models are useful, but stacking linear maps alone does not give a deeper kind of function.

To build richer networks, we need activation functions. An activation function is applied after a linear computation. It bends the model family so layers can combine into more than one straight boundary.

This chapter explains why that bend matters.

By the end, you should be able to:

• compute common activations by hand;
• explain what ReLU does;
• understand sigmoid and tanh as bounded nonlinearities;
• recognize saturation;
• read hidden units as small detectors;
• explain why stacked linear maps need nonlinearity between them.

The chapter is still framework-neutral. The goal is to understand the computation before it becomes a layer call.