Summary and Revision Notes
Key ideas
- An activation function transforms a pre-activation value.
- ReLU is
max(0, x). - ReLU is linear on each side of zero but nonlinear overall.
- Sigmoid maps values to
(0, 1). - Tanh maps values to
(-1, 1). - Saturation means the activation is nearly flat.
- Flat regions can make gradients small.
- Hidden units can be read as simple detectors.
- Stacked linear maps need nonlinear activations, or they collapse into one linear map.
Notation
| Notation | Meaning |
|---|---|
z | pre-activation value |
g(z) | activation applied to z |
a | activation output |
relu(x) | max(0, x) |
Common mistakes
- Treating activations as optional decoration.
- Forgetting that ReLU clips negative values to zero.
- Treating sigmoid outputs and raw scores as the same thing.
- Ignoring saturation when reasoning about gradients.
- Believing stacked linear layers become more expressive without nonlinearities.
Before moving on
You should be able to compute ReLU by hand, explain why sigmoid and tanh saturate, and show why two linear maps without an activation collapse into one linear map.