Review

Key Ideas

  • Sample: observed subset or draw from a larger population.
  • Estimator: rule that uses data to estimate an unknown quantity.
  • Train split: data used to fit model parameters.
  • Validation split: data used to choose settings.
  • Test split: data used for final reporting.
  • Bias: systematic error.
  • Variance: sensitivity to the sample.
  • Likelihood: plausibility of observed data under parameters.
  • Maximum likelihood: choosing parameters with highest likelihood.
  • Confidence interval: range expressing sampling uncertainty.
  • Hypothesis test: check whether evidence is surprising under a null assumption.
  • Cross-validation: repeated held-out evaluation across folds.
  • Resampling: repeated sampling from observed data.

Statistics is not a way to make weak evidence strong by naming it carefully. It is a way to keep the source, limits, and uncertainty of evidence visible.

Formulas to Remember

Sample mean:

xˉ=x1+x2+⋯+xnn\bar{x} = \frac{x_1 + x_2 + \cdots + x_n}{n}

Bayesian update shape:

posterior∝likelihood×prior\text{posterior} \propto \text{likelihood} \times \text{prior}

Maximum likelihood:

θ∗=arg⁡max⁡θL(θ)\theta^* = \arg\max_\theta L(\theta)

Interval width:

width=upper−lower\text{width} = \text{upper} - \text{lower}

Unnormalized Bayesian score:

score=prior weight×likelihood weight\text{score} = \text{prior weight} \times \text{likelihood weight}

Checks Before You Move On

  • Treating validation data as if it were test data.
  • Treating a test set as final after repeatedly checking it during development.
  • Reporting a point estimate without uncertainty when the difference is small.
  • Assuming a large dataset is representative.
  • Assuming maximum likelihood proves a model is true.
  • Treating a hypothesis test as a complete practical decision.
  • Treating a large p-value as proof that two systems are exactly the same.
  • Forgetting that resampling explores variation in the observed data, not in missing populations.
  • Treating dataset size as a substitute for coverage.
  • Forgetting that any score used to choose a model is development evidence.
  • Reading maximum likelihood as proof that a model family is true.
  • Reading a confidence interval as a promise about one future example.
  • Reading a confidence interval as protection against collection bias or distribution shift.
  • Fitting preprocessing on all folds before cross-validation.
  • Resampling dependent rows as if they were independent sampling units.
  • Reading a p-value as the probability that the null hypothesis is true.

Reading Model Reports

When reading an experiment, keep four questions nearby:

  • What sample or dataset produced the number?
  • Was the number used for fitting, choosing, or final reporting?
  • How much uncertainty or sample sensitivity is visible?
  • Is the difference practically meaningful, or only numerically different?
  • Did the reported number influence the choices that led to the report?

Mental Model

Statistics is the habit of asking how much evidence a number really carries.