Empirical Risk and Reduction

A loss first judges one example. Training needs one objective for many examples.

For a dataset of n input-target pairs (x_i, y_i), let the model with parameters theta predict f_theta(x_i). If L is the per-example loss, the empirical risk is its average over the observed dataset:

R_hat(theta) = (1 / n) sum_i L(f_theta(x_i), y_i)

“Empirical” means measured on the examples we actually have. It is not the unknown average loss over every future example the model might encounter.

One Dataset Objective by Hand

Suppose three examples have squared errors:

[1, 9, 2]

The common mean reduction gives:

(1 + 9 + 2) / 3 = 4

The sum reduction gives 12, while none keeps [1, 9, 2]. Here each entry is already one loss per example. These reductions contain the same per-example information, but their numerical scale differs. That scale matters later because gradients inherit it.

none -> one loss per example
sum  -> total loss across examples
mean -> average loss across examples

This book uses the mean unless a lesson says otherwise. Averaging makes losses more comparable across batch sizes. It does not make the dataset representative of future data; sampling and generalization remain separate questions.