ReLU
ReLU means rectified linear unit.
Its formula is:
relu(x) = max(0, x)
If the input is negative, ReLU returns 0.
If the input is positive, ReLU returns the input unchanged.
relu(-3) = 0
relu(0) = 0
relu(4) = 4
ReLU is simple, fast, and widely used. It also creates sparsity: some hidden units output exactly zero for a given input.
Why it helps
ReLU is linear on each side of zero, but not linear overall. The bend at zero is the important part. It lets a network combine straight pieces into richer shapes.
What is relu(7)?
Compute it first, then check your number.
HintPositive input
ReLU returns x when x > 0.
SolutionWork it out
Since 7 is positive, relu(7) = 7.
Apply ReLU to [-2, 0, 3, 5]. How many outputs are greater than zero?
Compute it first, then check your number.
HintClip first
The ReLU output is [0, 0, 3, 5].
SolutionWork it out
ReLU maps [-2, 0, 3, 5] to [0, 0, 3, 5]. The positive outputs are 3
and 5, so there are 2.