LLM Primer

Expectation

Expectation is the probability-weighted average of a random variable.

For a discrete random variable:

E[X]=∑xxP(X=x)E[X] = \sum_x xP(X=x)
12345E[X]weighted average
Expectation is the long-run weighted average of a random variable.

Example

Suppose:

X = 0 with probability 0.25
X = 4 with probability 0.75

Then:

E[X]=0(0.25)+4(0.75)=3E[X] = 0(0.25) + 4(0.75) = 3

The expected value is not necessarily the value you will see on one trial. It is the long-run average under the distribution.

In this example, X is never equal to 3. It is either 0 or 4. The expectation is still 3 because it is an average, not necessarily a possible outcome.

Expected Loss

If L is a random loss value, then expected loss is:

E[L]E[L]

This means average loss under the data distribution. In practice, we estimate it with a dataset or a mini-batch.

ML Reading

Many training objectives are expectations.

For example, expected loss means average loss under the data distribution. Since we usually do not know the full distribution, we estimate it with a dataset or mini-batch.

That estimate is itself noisy. A mini-batch loss is a small sample estimate of expected loss, not the exact expected loss over the whole data-generating process.

MATH-C07-T06-001Exercise: Compute expectation

Suppose X = 1 with probability 0.4 and X = 3 with probability 0.6.

What is E[X]?

Compute it first, then check your number.

Hint

Compute 1(0.4) + 3(0.6).

Solution

E[X] = 1(0.4) + 3(0.6) = 0.4 + 1.8 = 2.2. Each possible value contributes in proportion to its probability, so more likely values pull the average more.

MATH-C07-T06-002Exercise: Expectation may not be observed

Suppose X = 0 with probability 0.25 and X = 4 with probability 0.75.

Is E[X] = 3 one of the possible observed values?

Answer it first, then check.

Hint

The possible values are only 0 and 4.

Solution

No. The expectation is 3, but the random variable only takes values 0 and 4.

MATH-C07-T06-003Exercise: Compute expected loss

A loss is 2 with probability 0.5 and 6 with probability 0.5.

What is the expected loss?

Compute it first, then check your number.

Hint

Compute 2(0.5) + 6(0.5).

Solution

E[L] = 2(0.5) + 6(0.5) = 1 + 3 = 4. This is the probability-weighted average loss under the two possible cases.

MATH-C07-T06-004Exercise: Interpret expectation

Is expectation a probability or an average value?

Answer it first, then check.

Hint

Expectation weights values by their probabilities.

Solution

Expectation is an average value under uncertainty, not a probability. It can be larger than 1, and it has the units of the random variable.

MATH-C07-T06-005Exercise: Mini-batch estimate

Enter 1 if a mini-batch average loss is an estimate of expected loss, not usually the exact expected loss.

Compute it first, then check your number.

Hint

Ask whether a small batch contains every possible example.

Solution

Enter 1. A mini-batch average is computed from sampled examples, so it estimates expected loss rather than exactly equaling it in general.

Before Moving On

Expectation is average value under uncertainty.