Chapter 4
Activations and Nonlinearity
Activation functions, ReLU, sigmoid, tanh, saturation, hidden-unit detectors, and why depth needs nonlinear transforms.
What this chapter does
Activations are what let depth matter. This chapter shows how nonlinear transforms bend linear scores, how ReLU and bounded activations behave, and why stacked linear maps need activations between them.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Why nonlinear activations belong between linear model pieces.
- 02What Activation Functions Do
Pre-activations, activation outputs, and the role of nonlinear transforms.
- 03ReLU
The simple max(0, x) activation and why its bend matters.
- 04Sigmoid and Tanh
Bounded nonlinearities, historical context, and why their ranges matter.
- 05Saturation
Flat activation regions and why they can weaken gradient signals.
- 06Piecewise-Linear Behavior
How ReLU networks build richer functions from straight pieces.
- 07Hidden Units as Detectors
Reading hidden units as weighted evidence plus nonlinear response.
- 08Why Stacked Linear Maps Need Nonlinearity
Why depth only becomes expressive when activations interrupt linear maps.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What Chapter 4 established before multilayer perceptrons.
Summary and Revision NotesA compact review of activations, saturation, hidden units, and nonlinearity.
ExercisesChapter-level practice for activations and nonlinearity.
HintsLow-spoiler nudges for the Chapter 4 exercises.
SolutionsExplained solutions for the Chapter 4 exercises.
Before moving on
- Compute ReLU and identify activation outputs.
- Explain sigmoid, tanh, and saturation.
- Read hidden units as detector-like computations.
- Explain why stacked linear layers need nonlinear activations.
Where this leads
- Multilayer Perceptrons
- Backpropagation Through Networks
- Training stability