Activation Statistics

Activation statistics describe the values inside the network.

Useful summaries include:

• mean
• standard deviation
• minimum and maximum
• fraction of zeros
• fraction of non-finite values

These summaries tell whether information is flowing. A layer with almost constant activations may not be useful. A layer with huge activations may cause instability. A ReLU layer with mostly zeros may be silent.

For a small activation vector [0, 0, 2, 4], the fraction of zeros is:

2 / 4 = 0.5

Activation statistics are not a replacement for task metrics. They explain what is happening inside the run.