Chapter 11

Regularization and Generalization

Training loss, validation loss, overfitting, underfitting, weight decay, dropout, early stopping, data augmentation, and capacity.

Subject
Deep Learning
Lessons
7 lessons
Practice
21 checks
Review
3 pages

Introduction

A model can fit training data and still fail on new data.

That is the central problem of generalization. Training loss tells us how well the model fits examples it learned from. Validation loss tells us whether the learned pattern seems to hold on examples held out from training.

train lossvalidation losstraining timeloss
Overfitting often appears when training loss keeps improving while validation loss gets worse.

Regularization is a family of techniques that discourages brittle fits. It does not make a model automatically correct. It adds pressure toward simpler, more stable, or less overly specific solutions.

This chapter teaches the first practical signs: underfitting, overfitting, weight decay, dropout, early stopping, data augmentation, and the relationship between capacity and dataset size.

Lessons

Read these in order.

The chapter opening gives the main idea. Move through these lessons next; each page reuses ideas from the pages before it.

  1. 01
    Train Loss Versus Validation Loss

    Reading training and validation losses as answers to different questions.

  2. 02
    Overfitting and Underfitting

    Diagnosing whether a model is failing to fit or fitting too specifically.

  3. 03
    Weight Decay

    Penalizing unnecessarily large weights to reduce brittle fits.

  4. 04
    Dropout

    Dropout as a training-time perturbation that discourages fragile hidden-unit reliance.

  5. 05
    Early Stopping

    Using validation behavior to stop before overfitting deepens.

  6. 06
    Data Augmentation

    Adding safe variation to training examples while preserving targets.

  7. 07
    Capacity and Dataset Size

    Reading model flexibility together with dataset size, noise, and evidence.

  1. •
    Conclusion

    What regularization adds before evaluation and debugging.

  2. •
    Review

    A compact review of overfitting, underfitting, weight decay, dropout, early stopping, augmentation, and capacity.

  3. •
    Exercises

    Chapter-level practice for regularization and generalization.

You are ready when

  • Distinguish training loss from validation loss.
  • Recognize underfitting and overfitting patterns.
  • Compute simple weight-decay and capacity ratios.
  • Explain how dropout, early stopping, and augmentation support generalization.

Where this leads

  • Evaluation and Debugging
  • Embeddings and Learned Representations
  • Convolutional Neural Networks

Chapter progress