Exercises
These exercises review the chapter as a whole. Some are numerical; others check whether you can read an experiment without overclaiming.
A dataset has 200 examples. The train split has 140 examples and the
validation split has 30 examples.
How many examples remain for the test split?
Compute it first, then check your number.
Compute the mean of:
Compute it first, then check your number.
A confidence interval runs from 0.62 to 0.74.
What is its width?
Compute it first, then check your number.
In 4-fold cross-validation, how many validation runs are made?
Compute it first, then check your number.
A prior weight is 0.3 and a likelihood weight is 0.6.
What is the unnormalized posterior weight?
Compute it first, then check your number.
Should a test split be checked repeatedly while choosing the model?
Answer it first, then check.
A model has high training error and high validation error.
Which word is the better first diagnosis: bias or variance?
Answer it first, then check.
Does a higher likelihood prove that a model is the true explanation?
Answer it first, then check.
In bootstrap resampling, can the same example appear more than once in one resampled dataset?
Answer it first, then check.
A hypothesis test finds evidence of a difference between two models.
Does that automatically mean the difference is large enough to matter in deployment?
Answer it first, then check.
Enter 1 if a very large dataset can still be weak evidence for a population
it does not cover well.
Compute it first, then check your number.
Enter 1 if a reasonable estimator can still give a misleading estimate on one
unlucky sample.
Compute it first, then check your number.
Enter 1 if maximum likelihood can choose the best parameter among candidates
without proving that the whole model family is true.
Compute it first, then check your number.
Enter 1 if a confidence interval reports uncertainty in an estimate rather
than promising the outcome of one future example.
Compute it first, then check your number.
Enter 1 if cross-validation used to choose a model may still need a separate
untouched test set for final reporting.
Compute it first, then check your number.
Next
Use the hints only after you have tried the exercises.