Scaling Laws Are Empirical Fits within a Measured Regime

A scaling law summarizes measurements made while selected quantities change. It is not a law that guarantees every larger model will behave as an extrapolated line predicts.

Fit an Illustrative Power Law

Suppose loss follows the assumed form

where $C$ is compute. For two illustrative points $(1,2.0)$ and $(4,1.5)$,

The fitted line gives $L(16)=1.125$. This is an unvalidated extrapolation. Two points determine a line in log space but provide no independent evidence that the line continues.

State What Was Held Fixed

A useful scaling statement records:

• model family and parameter range;
• dataset and token allocation;
• objective, optimizer, schedule, and training procedure;
• compute definition and measurement method;
• loss or task metric;
• fitted range and uncertainty.

A lower pretraining loss is not identical to every downstream capability, and a trend inside the measured range does not establish behavior outside it.

Kaplan et al. (2020) report empirical language-model scaling relationships under their experimental setup. Treat the reported exponents and conclusions as measurements with those boundaries.