Sanity Checks and Seeds
Sanity checks ask whether the training setup can pass simple tests.
Examples:
- can the model overfit a tiny batch?
- do labels and predictions have matching shapes?
- does loss decrease on an easy synthetic task?
- do metrics change when labels are shuffled?
- are results similar across a few random seeds?
A random seed controls random choices such as initialization and shuffling. If one seed gives a great result and several others fail, the result may be fragile.
Seed sensitivity is not automatically bad. It is information. Small datasets and unstable training can make results vary more.
The strongest habit is to make claims from repeated evidence, not from one lucky run.
Exercise: Tiny batch check
Enter 1 if overfitting a tiny batch can be a useful sanity check, or 2 if it is never useful.
Compute it first, then check your number.
Exercise: Seed average
Validation accuracies from three seeds are 0.70, 0.76, and 0.72. What is their average?
Compute it first, then check your number.