Review

Key ideas

• Training loss and validation loss answer different questions.
• Underfitting means the model is not fitting training data well.
• Overfitting means training fit does not transfer well.
• Weight decay adds pressure against large weights.
• Dropout is a training-time perturbation.
• Early stopping uses validation behavior to stop before overfitting deepens.
• Data augmentation should preserve the target.
• Capacity must be judged with dataset size, noise, and validation evidence.

Common formulas

total_loss = data_loss + lambda * sum(weights^2)
weight_decay_penalty = lambda * w^2
parameters_per_example = number_of_parameters / number_of_examples

Common mistakes

• Treating low training loss as proof of useful learning.
• Ignoring the validation curve until the end.
• Applying augmentation that changes the target without updating the target.
• Assuming regularization fixes bad data or a wrong task definition.
• Choosing the biggest model before checking the amount and quality of evidence.

Before moving on

You should be able to read train and validation curves, recognize underfitting and overfitting, compute a simple weight-decay penalty, and explain why regularization is about transfer, not only fitting.