Capacity and Dataset Size
Capacity is the range of patterns a model can represent.
A high-capacity model can fit complex patterns. That is useful when the data supports those patterns. It is risky when the dataset is small, noisy, or mismatched with the task.
Dataset size does not decide everything, but it changes the risk. A model with many flexible parameters can memorize a small dataset more easily than a large, varied one.
Regularization manages this tension:
more capacity: can fit richer patterns
more regularization: discourages brittle fits
more useful data: gives better evidence about real patterns
The best model is not always the biggest model. The best model is the one whose capacity, data, training procedure, and evaluation evidence fit the problem.
Enter 1 if high capacity always guarantees better validation performance, or 2 if high capacity can overfit when evidence is weak.
Compute it first, then check your number.
A model has 1,000 parameters and the training set has 100 examples. How many parameters per training example is that?
Compute it first, then check your number.