Objectives and Loss Functions
An objective is the quantity we want to optimize.
A loss function is an objective we usually want to minimize.
For a prediction yhat and target y, a simple squared loss is:
The loss is a number. That sounds simple, but it is a major design choice.
Once behavior becomes a number, we can compare two models, compute gradients, and update parameters.
This number is not the whole task. It is a proxy. A proxy is useful because the optimizer can work with it, but it can also hide what the reader or user actually cares about.
Why Loss Exists
A model does not know what "better" means unless we define it.
The loss function turns model behavior into a number that can be compared, differentiated, and reduced.
For example, if a prediction is 5 and the target is 3, squared loss gives
4. If another prediction is 4, squared loss gives 1. Under this loss, the
second prediction is better.
Objective Versus Metric
A training loss is the number the optimizer directly uses.
A metric is often the number humans care about when evaluating a model.
They can be different. A classifier may train with cross-entropy while people also report accuracy. Cross-entropy gives smoother gradient information than a simple correct-or-incorrect count.
That is why a training loss can improve while a reported metric stays flat for a while. The optimizer may be receiving smaller, smoother signals than the final metric reveals.
ML Reading
Different tasks use different losses.
- regression often uses squared or absolute error
- classification often uses cross-entropy
- reinforcement learning uses objectives built from rewards
Let yhat = 5 and y = 3.
Compute (yhat - y)^2.
Compute it first, then check your number.
Hint
Solution
(5 - 3)^2 = 2^2 = 4. The loss first measures the prediction error, then
squares it to produce the objective value.
Target y = 3. Prediction A is 5, prediction B is 4.
Under squared loss, which prediction has lower loss: A or B?
Answer it first, then check.
Hint
Compute (yhat - y)^2 for both predictions.
Solution
Prediction A has loss (5 - 3)^2 = 4. Prediction B has loss (4 - 3)^2 = 1.
Prediction B has lower loss.
Is the training loss always the same number as the metric humans report?
Answer it first, then check.
Hint
The lesson compares cross-entropy and accuracy.
Solution
No. The optimizer may use one loss for training while humans report a different metric for evaluation.
If the objective is badly chosen, can optimization improve the wrong thing?
Answer it first, then check.
Hint
Optimization follows the objective it is given.
Solution
Yes. Optimization can faithfully reduce a number even when that number does not match the behavior we actually want.
Enter 1 if a loss function is usually a trainable proxy for the behavior we
want, not the full behavior itself.
Compute it first, then check your number.
Hint
Ask whether one number can contain every human judgment about model behavior.
Solution
Enter 1. The loss is the number the optimizer follows. It should be aligned
with the desired behavior, but it is still a proxy.
Before Moving On
The loss function defines what the training process tries to improve.