Bias and Variance as a Diagnostic
In this context, bias and variance describe two ways predictions can be wrong. This use of “bias” is about systematic prediction error, not social or measurement bias in a dataset.
- High bias: the learning procedure is too restricted or insufficiently trained to capture an important pattern.
- High variance: the fitted model changes too much when the training sample changes.
These tendencies often resemble underfitting and overfitting:
high training error -> possible high bias
low training error, worse validation -> possible high variance
The mapping is diagnostic, not a proof. Optimization failure, label noise, distribution shift, leakage, or a mismatched metric can produce similar curves. The formal squared-error decomposition defines bias and variance through an expectation over repeated training samples and separates irreducible noise. One train/validation split cannot measure that decomposition; this lesson uses the terms only to organize hypotheses that must be tested.
See Variance Across Training Sets
Suppose the same procedure is trained on three sampled training sets, then all three fitted models predict the same input:
predictions = [4, 10, 7]
mean prediction = 7
Their predictions vary substantially around the mean. That instability is the
practical meaning of variance here. If instead all three predict near 4 when
the correct target is 9, their predictions are stable but systematically
wrong—a high-bias pattern.
Useful responses differ:
possible high bias:
verify optimization, improve features/model capacity, or reduce excessive regularization
possible high variance:
add representative data, strengthen justified regularization,
simplify the model, or average multiple fits
More capacity can reduce bias while increasing variance, but modern deep networks do not obey a single monotonic rule based only on parameter count. Use held-out experiments rather than treating the phrase “bias–variance trade-off” as an automatic prescription.
Several retrained models make nearly the same prediction, but all are far from the target. Which diagnosis is more plausible?
Select one choice, then check.
HintAsk whether the fitted result changes
Variance describes sensitivity across sampled training sets.
SolutionSystematic error
This is more consistent with high bias: the procedure repeatedly produces a similar but wrong prediction. High variance would show larger changes across retrained models.
The same input receives predictions [4, 10, 7] from models trained on three
sampled training sets. What is their mean prediction?
Compute it first, then check your number.
HintAverage the fits
Add the three predictions and divide by three.
SolutionCenter the predictions
(4 + 10 + 7) / 3 = 21 / 3 = 7. Their spread around 7 is evidence of
sensitivity to the sampled training data.
Does low training loss and high validation loss prove that model capacity is the only cause?
Select one choice, then check.
HintPattern versus proof
Several data and evaluation problems can widen the gap.
SolutionKeep alternative hypotheses
No. The pattern is consistent with high variance or overfitting, but data leakage, distribution mismatch, noisy validation estimates, or inconsistent evaluation can create similar evidence.