Dead ReLU Detection

A ReLU unit outputs zero when its input is negative.

That is normal. But if a ReLU unit outputs zero for almost every example over many steps, it may be effectively dead. It contributes little signal and may receive little useful gradient.

Dead ReLU detection is usually simple: track the fraction of zero outputs per layer or unit.

zero_fraction = number_of_zero_outputs / total_outputs

If the zero fraction is extremely high, inspect initialization, learning rate, bias values, input scale, and optimizer behavior.

The goal is not to force every ReLU active. Sparsity can be useful. The warning is a layer or unit that is silent when it should be learning.