Weight Decay

Weight decay discourages unnecessarily large weights.

One way to read it is as a penalty added to the loss:

total_loss = data_loss + lambda * sum(weights^2)

The penalty does not say large weights are always wrong. It says large weights should be paid for. If two solutions fit the data similarly, weight decay prefers the one with smaller weights.

Weight decay adds pressure against unnecessarily large weights.

This can reduce brittle fits because the model is discouraged from using extreme parameter values unless they clearly help.

For a single weight w = 3 and lambda = 0.1, the penalty term is:

0.1 * 3^2 = 0.9

Exercise: Weight decay penalty

Let w = 4 and lambda = 0.5. What is lambda * w^2?

Your answer

Compute it first, then check your number.

Exercise: Penalty direction

Enter 1 if weight decay encourages larger weights, or 2 if it discourages unnecessarily large weights.

Your answer

Compute it first, then check your number.