LLM Primer

Estimators

An estimator is a rule for using data to estimate an unknown quantity.

The sample mean is an estimator. It uses observed numbers to estimate a typical value.

sample dataruleestimatesample changes, estimate moves
An estimator is a rule that turns sample data into an estimate.

Example

For the values:

2, 4, 62,\ 4,\ 6

the sample mean is:

2+4+63=4\frac{2 + 4 + 6}{3} = 4

In ML

Validation accuracy estimates future accuracy.

Average loss on a dataset estimates expected loss on the population we care about.

These estimates are useful, but they are still estimates. They can move when the sample changes.

This is the central caution. An estimator may be reasonable and still be wrong for a particular sample.

That is not a contradiction. Statistics does not promise that every sample gives the right number. It asks whether the rule behaves well across repeated samples from the population we care about.

Estimator Versus Estimate

The estimator is the rule.

The estimate is the number produced after applying the rule to data.

For example, "take the sample mean" is an estimator. The number 5 is an estimate for the sample 2, 4, 6, 8.

MATH-C10-T04-001Exercise: Compute a sample mean

What is the mean of 2, 4, 6, 8?

Compute it first, then check your number.

Hint

Add the four values, then divide by 4.

Solution
2+4+6+84=204=5\frac{2 + 4 + 6 + 8}{4} = \frac{20}{4} = 5

The estimator is the rule "take the mean." The estimate from this sample is the number 5.

MATH-C10-T04-002Exercise: Rule or result

Is "take the sample mean" an estimator or an estimate?

Answer it first, then check.

Hint

The rule is the estimator.

Solution

"Take the sample mean" is an estimator because it is a rule for producing an estimate.

MATH-C10-T04-003Exercise: Moved sample

Can an estimate change when the sample changes?

Answer it first, then check.

Hint

The lesson says estimates can move when the sample changes.

Solution

Yes. Since an estimate is computed from sample data, changing the sample can change the estimate.

MATH-C10-T04-004Exercise: Validation accuracy

Is validation accuracy an estimate of future accuracy?

Answer it first, then check.

Hint

The validation set is a sample used to estimate future behavior.

Solution

Yes. Validation accuracy estimates how the model may perform on future examples, but it is still an estimate.

MATH-C10-T04-005Exercise: Reasonable but wrong

Enter 1 if a reasonable estimator can still give a poor estimate on one unlucky or unrepresentative sample.

Compute it first, then check your number.

Hint

Think about validation accuracy computed on a small unusual validation set.

Solution

Enter 1. An estimator is a rule. It can be useful overall while still giving a misleading estimate on one sample.

Before Moving On

An estimator is not the truth. It is a rule for making a disciplined guess.