Chapter 10

Initialization, Scale, and Normalization

Weight initialization, activation scale, gradient scale, input normalization, batch normalization, layer normalization, RMSNorm, and scale diagnostics.

Subject
Deep Learning
Lessons
7 lessons
Practice
21 checks
Review
3 pages

Introduction

A training loop can be correct and still fail.

Sometimes the problem is not the architecture, optimizer, or dataset. Sometimes the numbers moving through the network are badly scaled. Activations may grow until they overflow, shrink until they carry little signal, or sit in saturated regions where gradients are small.

inputlayer 1layer 2layer 3lossrepeated layers can amplify or shrink signalsscale affects both activations and gradients
If values grow or shrink each layer, deep networks can become unstable before learning begins.

Initialization, scale, and normalization are practical tools for keeping signals usable. They do not replace learning. They make learning possible.

This chapter asks a simple question: before blaming the model, are the values in a range where gradients can do useful work?

Lessons

Read these in order.

The chapter opening gives the main idea. Move through these lessons next; each page reuses ideas from the pages before it.

  1. 01
    Why Scale Matters

    How repeated layers can amplify or shrink activations and gradients.

  2. 02
    Weight Initialization

    Starting parameter scale, symmetry breaking, and stable first steps.

  3. 03
    Activation Scale

    Inspecting intermediate values so later layers receive usable signals.

  4. 04
    Gradient Scale

    How gradient magnitude and learning rate combine into update size.

  5. 05
    Input Normalization

    Recentering and rescaling raw features before they enter the model.

  6. 06
    Layer, Batch, and RMS Normalization

    Normalization inside networks, with batch norm, layer norm, and RMSNorm roles.

  7. 07
    Diagnosing Scale Problems

    Practical signs of exploding, vanishing, saturated, or non-finite values.

  1. •
    Conclusion

    What scale controls make possible before generalization questions.

  2. •
    Review

    A compact review of initialization, activation scale, gradient scale, normalization, and diagnostics.

  3. •
    Exercises

    Chapter-level practice for scale, initialization, and normalization.

You are ready when

  • Explain why repeated layers can amplify or shrink signals.
  • Recognize initialization and activation-scale problems.
  • Compute simple normalized values and update sizes.
  • Name the roles of input normalization, batch norm, layer norm, and RMSNorm.

Where this leads

  • Regularization and Generalization
  • Evaluation and Debugging
  • Residual, Skip, and Gated Connections

Chapter progress