Introduction

Statistics is the discipline of learning from limited data without pretending that limited data is the whole truth.

Machine learning needs this because a model is not judged only by how well it fits the examples it has seen. It is judged by how well its learned pattern survives new examples.

The Main Idea

Statistics separates three questions:

• What data did we observe?
• What pattern are we estimating from it?
• How much should we trust that estimate?

In ML, these questions appear in simple places: train and test splits, validation curves, likelihood, confidence intervals, and error bars.

Optimization asked, "How do we reduce the loss?"

Statistics asks a different question:

Should we trust what that reduced loss seems to tell us?

That is why statistics belongs right after optimization. A model can fit its training data while still failing on new data, hidden subgroups, or future examples.

What This Chapter Covers

This chapter covers samples, splits, estimators, bias and variance, likelihood, maximum likelihood, simple Bayesian updating, confidence intervals, hypothesis tests, cross-validation, and resampling.

The goal is not to become a statistician. The goal is to read ML experiments with better judgment.

Read the chapter with two questions nearby:

• What evidence was used?
• What uncertainty remains?

Before Moving On

Keep one sentence nearby:

Data is evidence, not certainty.