Train, Validation, and Test Splits
The three common splits have different jobs.
Training data updates parameters. Validation data guides decisions during development. Test data is held back for a final estimate after model choices are mostly fixed.
If the test set is used repeatedly to choose models, it stops being a clean final check. It becomes part of development.
The split discipline is not bureaucracy. It protects you from believing a model generalizes because you accidentally tuned it to the evaluation set.
Enter 1 for train, 2 for validation, or 3 for test: this split is used for parameter updates.
Compute it first, then check your number.
Enter 1 for train, 2 for validation, or 3 for test: this split should be saved for the final estimate.
Compute it first, then check your number.