Parameter Histograms

Parameter histograms show how weights are distributed.

They help answer questions such as:

  • are weights centered near zero?
  • are they spreading over training?
  • are many values stuck at exactly zero?
  • are a few values becoming extreme?
parameter histogramvalues, not only final accuracy, deserve inspection
Parameter histograms expose whether weights are centered, spreading, collapsing, or becoming extreme.

A histogram does not tell whether the model is good by itself. It tells whether the parameters look plausible or suspicious.

If a run suddenly produces extreme weights, connect that observation to the training loop: learning rate, gradients, loss scale, and normalization may all be involved.

Exercise: Histogram role

Enter 1 if a parameter histogram directly proves generalization, or 2 if it is an internal diagnostic.

Compute it first, then check your number.

Exercise: Extreme weight count

A histogram shows 3 weights with magnitude above 100 out of 60 total weights. What fraction is extreme?

Compute it first, then check your number.