Layer, Batch, and RMS Normalization
Normalization can also happen inside a network.
Batch normalization uses statistics across a batch. It was important in earlier deep-learning practice and remains useful in many settings, especially convolutional networks.
Layer normalization uses statistics within each example's features. It is important in sequence models and transformers because it does not depend on batch statistics in the same way.
RMSNorm is a related transformer-era normalization. It rescales by root-mean-square magnitude and does not subtract the mean. It is often used because it is simpler while still controlling scale.
The details differ, but the shared idea is stable:
normalization keeps intermediate values in a range where learning behaves better
This chapter is not asking you to memorize implementation formulas. It is asking you to recognize why these layers exist.
Enter 1 if normalization mainly changes the model's final task, or 2 if it mainly helps keep intermediate values well scaled.
Compute it first, then check your number.
Enter 1 for batch normalization, or 2 for layer normalization: this method uses statistics across a batch.
Compute it first, then check your number.