Train, Validation, and Test Splits
A train split is used to fit model parameters.
A validation split is used to choose settings: model size, learning rate, regularization, stopping point, and similar decisions.
A test split is used at the end to report performance.
Why Split?
If the same examples are used to train, tune, and report, the reported number can become too optimistic.
The model may not have learned the underlying pattern. It may have learned the quirks of the data and choices made during tuning.
The split gives each part a job:
- train: fit parameters
- validation: choose settings and stopping points
- test: report a final estimate after choices are made
If the test split is checked repeatedly during tuning, it slowly becomes part of the tuning process. That is called leakage in spirit, even when no file is copied by mistake.
The deeper rule is simple: any data that influences a choice is no longer independent evidence for the final report. It may not touch the optimizer, but it can still shape the experiment.
Small Example
A dataset has 100 examples. We use 70 for training, 15 for validation, and 15 for testing.
How many examples are not used for training?
Compute it first, then check your number.
Hint
Add validation and test examples.
Solution
The non-training examples are 15 validation examples plus 15 test examples:
Which split is used to fit model parameters: train, validation, or test?
Answer it first, then check.
Hint
Training fits parameters.
Solution
The train split is used to fit model parameters. Validation and test data should not play that same role.
Which split is used to choose settings such as learning rate or stopping point:
train, validation, or test?
Answer it first, then check.
Hint
Validation is for choosing among options.
Solution
The validation split is used to choose settings and stopping points. It guides development choices without being the final report split.
If you repeatedly use the test split while tuning choices, can the final report become too optimistic?
Answer it first, then check.
Hint
Repeated looks turn the test split into part of the decision process.
Solution
Yes. Repeatedly checking the test split during tuning can make the final test number too optimistic.
Enter 1 if a test score can become less trustworthy when it influences model
choices, even if the model was not directly trained on the test examples.
Compute it first, then check your number.
Hint
Ask whether the test result affected what was chosen next.
Solution
Enter 1. If the test score guides model selection, it becomes part of the
development process and the final report can become too optimistic.
Before Moving On
Use the test split sparingly. Each repeated look can turn it into another validation set.