Regularization
Regularization adds a preference to the objective.
For example, an objective may combine data loss with a penalty:
The coefficient lambda controls how strongly the penalty matters.
If lambda = 0, the penalty is ignored. If lambda is large, the optimizer may
prefer lower-penalty parameters even when the data loss is slightly worse.
Why Add a Penalty?
The data loss measures fit.
The penalty can discourage unwanted parameter patterns, such as very large weights.
This can help reduce overfitting and improve generalization.
Regularization does not guarantee generalization. It changes the preference of the training objective. Whether that preference helps must be checked on data not used for fitting.
ML Reading
Regularization is not one technique. It is a family of ways to shape the solution we prefer.
Examples include weight decay, dropout, early stopping, and data augmentation.
Read regularization as a bias we choose on purpose. It says, "among the solutions that fit the data, prefer this kind of solution." That preference may help, but it should be checked rather than trusted blindly.
If data loss = 5 and penalty = 2, what is their sum?
Compute it first, then check your number.
Hint
Solution
5 + 2 = 7. This is the simplest regularized-objective pattern: combine the
fit term with the penalty term.
If data loss = 5, penalty = 2, and lambda = 0.5, what is
data loss + lambda * penalty?
Compute it first, then check your number.
Hint
Compute 5 + 0.5 * 2.
Solution
5 + 0.5 * 2 = 5 + 1 = 6. The coefficient lambda controls how much of the
penalty enters the objective. Here only half of the penalty is added to the data
loss.
If lambda = 0, does the penalty affect the objective?
Answer it first, then check.
Hint
The penalty is multiplied by lambda.
Solution
No. When lambda = 0, the penalty term contributes 0. The objective then
reduces to the data loss alone.
Does regularization guarantee better generalization in every case?
Answer it first, then check.
Hint
The lesson says the preference must be checked on held-out data.
Solution
No. Regularization changes the training preference, but it does not guarantee better generalization in every case.
Enter 1 if regularization changes the preferred solution but does not prove
that the model will generalize better.
Compute it first, then check your number.
Hint
Ask whether a penalty is the same as validation evidence.
Solution
Enter 1. Regularization shapes the objective. Whether it helps generalization
must be checked with data that was not used for fitting.
Before Moving On
Regularization changes what kind of solution optimization prefers.