Diagnosing Scale Problems

Scale problems often leave visible traces.

Look for:

• loss becomes NaN or infinite
• loss explodes in the first few steps
• gradients are exactly zero or extremely tiny
• gradients are huge in early layers
• ReLU activations are mostly zero
• sigmoid or tanh activations are stuck near their limits
• training changes dramatically when inputs are normalized

These signs do not prove one exact cause. They tell you where to inspect.

A useful debugging habit is to log small summaries:

activation mean and standard deviation
gradient norm
parameter norm
minimum and maximum values
fraction of non-finite values

You do not need a large system to learn this habit. Even a tiny NumPy-style model can print these values after each step.