Bias and Variance
Bias and variance describe two ways an estimate can go wrong.
Bias is systematic error.
Variance is sensitivity to the particular sample.
Working Intuition
A biased estimator tends to miss in the same direction.
A high-variance estimator may sometimes be close and sometimes be far, depending on which data it saw.
Bias is about the center of many attempts. Variance is about how spread out those attempts are.
An estimator can have both problems. It can be consistently off and unstable.
This is why one train-validation split is only a clue. To see bias and variance clearly, imagine repeating the experiment with new samples from the same population.
In ML
An underfit model often has high bias. It is too simple to represent the pattern.
An overfit model often has high variance. It reacts too much to the training sample.
If training error is high and validation error is high, the model may be too limited. If training error is low but validation error is high, the model may be reacting too much to the training sample.
A model performs poorly on both training data and validation data.
Enter 1 if this suggests high bias more than high variance.
Compute it first, then check your number.
Hint
If it cannot fit even the training data well, it is probably too limited.
Solution
Enter 1. Poor training and validation performance often suggests high bias or underfitting.
A model performs very well on training data but poorly on validation data.
Does this suggest high variance more than high bias?
Answer it first, then check.
Hint
The model fits the training sample but does not transfer well.
Solution
Yes. Strong training performance with poor validation performance often suggests high variance or overfitting.
Which word names systematic error: bias or variance?
Answer it first, then check.
Hint
Bias is about consistently missing in the same direction.
Solution
Bias names systematic error. It is about the center of many attempts missing in a consistent direction.
Which word names sensitivity to the particular sample: bias or variance?
Answer it first, then check.
Hint
Variance is about spread across attempts.
Solution
Variance names sensitivity to the particular sample. It is about how much the estimate moves across repeated samples.
Enter 1 if bias and variance are easiest to understand by imagining many
repeated samples, not just one result.
Compute it first, then check your number.
Hint
One result cannot show the center and spread of many attempts.
Solution
Enter 1. Bias and variance describe how estimates behave across repeated
samples or repeated training runs.
Before Moving On
Bias and variance are not insults. They are lenses for reading model behavior.