What Activation Functions Do

A linear layer produces a raw value:

z = x . w + b

An activation function transforms that value:

a = g(z)

The activation output a becomes the value passed to the next part of the model.

Without activation functions, a stack of linear layers collapses into another linear map. With activation functions, the model can build nonlinear patterns from simple pieces.

One scalar example

Suppose:

z = -3

A ReLU activation gives:

relu(z) = max(0, z)
        = max(0, -3)
        = 0

The activation changed the value. That change is what lets later layers receive a transformed signal rather than just another linear score.