Compute-Optimal Training Couples Model, Data, and Budget

Use a simplified compute ledger to compare parameter-token allocations, then retain data quality and empirical optimization boundaries.

A parameter count alone does not specify a training run. Under a fixed compute budget, a larger model usually permits fewer training tokens, while a smaller model permits more. The allocation can change the final loss.

Use a Simple Budget Model

For a dense Transformer training estimate, write

C\approx kND,

where $N$ is the parameter count, $D$ is the number of training tokens, and $k$ collects architecture and accounting constants. If $C/k=1200$ , then both allocations satisfy the same simplified budget:

Allocation	$N$	$D$	$ND$
A	100	12	1200
B	40	30	1200

The budget equation alone cannot tell which obtains lower loss. That requires an empirical relation involving model size, data, and the tested regime.

Keep Tokens and Data Quality Visible

Training tokens are not interchangeable units of information. Distribution, duplication, filtering, contamination, curriculum, modality, and tokenizer all matter. Repeating a small dataset is not automatically equivalent to observing new samples.

Q1. Complete a compute allocation

Under $C/k=2400$ and the approximation $C/k=ND$ , a model has $N=80$ . How many training tokens $D$ fit the simplified budget?

Your answer

Compute it first, then check your number.

Hint

Divide 2400 by 80.

Solution

D=2400/80=30

tokens in the simplified units.

Compare Studies without Blending Them

Hoffmann et al. (2022) re-examined compute-optimal allocation using experiments and analyses that differed from earlier scaling work. Preserve each study's models, data, compute definitions, fitted range, and objective. “More parameters is always better” and “one fixed token-to-parameter ratio is universal” are both too strong.