Weight Initialization

Initialization chooses the starting parameter values.

If all weights start at the same value, many units compute the same thing and receive similar gradients. The network has capacity on paper, but the units begin without useful diversity.

If weights are too small, activations can shrink layer by layer. If weights are too large, activations can grow or push nonlinearities into saturated regions.

too smallbalancedtoo largeinitialization chooses the starting spread of parameters
Too little spread can make units behave alike; too much spread can create unstable activations.

Good initialization gives the network a reasonable starting scale. It does not make the model trained. It only avoids making the first training step unnecessarily broken.

The exact initialization rule depends on the layer and activation. For this chapter, the main idea is enough: initialize with a spread that preserves useful signal through the network.

Exercise: Symmetry problem

Enter 1 if all hidden units starting with the same weights is usually helpful, or 2 if it can make units behave too similarly.

Compute it first, then check your number.

Exercise: Too-large start

Enter 1 for likely stable scale, or 2 for a warning sign: initial weights make activations grow from 1 to 20 to 400.

Compute it first, then check your number.