Review

Key ideas

• Correct training code can still fail because of bad scale.
• Repeated layers can amplify or shrink activations and gradients.
• Initialization sets the starting parameter scale.
• Identical initialization can make units behave too similarly.
• Input normalization recenters and rescales raw features.
• Batch normalization uses batch statistics.
• Layer normalization uses per-example feature statistics.
• RMSNorm controls magnitude without subtracting the mean.
• Scale diagnostics include activation statistics, gradient norms, parameter norms, and non-finite values.

Common formulas

normalized_x = (x - mean) / standard_deviation
update_size = learning_rate * gradient
repeated_scale = starting_scale * multiplier^layers

Common mistakes

• Blaming architecture before checking values.
• Assuming initialization trains the model.
• Treating normalization as a change to the task instead of a scale control.
• Looking only at loss while ignoring activations and gradients.
• Forgetting that too-small gradients can be as harmful as too-large gradients.

Before moving on

You should be able to explain why scale matters, compute a simple normalized value, identify exploding or vanishing signals, and name the role of initialization and normalization.