Initialization Sets the First Signal Scale

Before the first optimizer step, initialization determines the magnitudes of token records, attention scores, residual updates, logits, loss, and gradients. The model is structurally correct at many possible scales, but not every scale is equally easy to optimize.

The frozen experiment uses:

• model seed 7;
• zero-mean normal weights with standard deviation $0.1$;
• zero linear biases;
• LayerNorm scales initialized to 1 and offsets to 0;
• zero dropout.

These are one tested toy configuration, not general Transformer defaults.

Seed before Constructing the Model

Setting the seed after construction cannot reproduce parameters that have already been sampled. Data sampling and batch order use separate generators so changing one does not silently change the others.

Audit Distributions, Not One Convenient Number

For every parameter group, record at least shape, minimum, maximum, mean, and standard deviation. Then run one batch without updating and inspect:

With eight vocabulary entries, uniform next-token probabilities give $-\ln(1/8)=\ln 8\approx2.0794$ nats. The verified model begins near this value: 2.1898 on the sampled training corpus and 2.1918 on validation. Near-uniform initial predictions explain those values; the numbers do not prove that every internal scale is healthy.

Compare Scale while Holding Everything Else Fixed

An initialization ablation should change only the standard deviation. Keep the model seed, corpus, batch order, optimizer, and number of updates fixed. Measure initial logits, loss, residual norms, gradients, and later loss.

• Very small weights can produce weak distinctions and small updates in some paths.
• Very large weights can produce extreme scores, saturated probabilities, or unstable residual and gradient scales.
• Normalization can reduce some scale problems without making initialization irrelevant.

Do not describe one successful standard deviation as universally optimal.

Initialization Is the First Recorded Experiment State

Store the initialization rule and model seed in the configuration. A checkpoint can later preserve trained weights, but it cannot explain how an unrecorded run began.