Activation Scale
Activations are the values passed from one layer to the next.
Their scale matters because the next layer reads them as input. If activations are extremely large, the next layer receives large numbers. If activations are almost all zero, the next layer receives little usable signal.
Activation functions can make this worse or better. ReLU keeps positive values and zeros out negative values. Sigmoid and tanh can saturate: very large positive or negative inputs produce outputs near their limits, where derivatives are small.
In practice, inspecting activation statistics can reveal a lot:
mean: are values centered?
standard deviation: are values spread too little or too much?
minimum and maximum: are there extreme values?
fraction of zeros: are ReLU units mostly inactive?
These are not abstract diagnostics. They tell you whether signals are flowing.
A ReLU layer receives 10 values and outputs zero for 8 of them. What fraction of outputs are zero?
Compute it first, then check your number.
Enter 1 for likely saturated sigmoid inputs, or 2 for centered small inputs: values are mostly near 50 and -50.
Compute it first, then check your number.