Audit a Training Run Systematically

Diagnose data, target, mask, shape, scale, loss, gradient, optimizer, validation, and checkpoint failures in dependency order. Require controlled variables and narrow conclusions for every ablation.

Training joins many contracts. Diagnose them in dependency order so a later symptom is not mistaken for the first error.

1. Data and Targets

decode several input and target rows;
verify one-position shifting and sequence boundaries;
check vocabulary and context ranges;
confirm validation examples never enter updates;
record realized sequence counts.

An input copied as its own target creates an easier but wrong problem. A future token placed in the input can make loss fall while invalidating the model.

2. Architecture and Information Flow

assert all shapes and the 368-parameter ledger;
verify tied parameter identity;
intervene on future tokens and test earlier-logit invariance;
confirm training and evaluation use the intended model behavior.

Dropout is zero in this model, but calling train() and eval() remains good state discipline and matters when stochastic or stateful layers are introduced.

3. Forward Values and Loss

locate the first non-finite or extreme activation;
inspect residual, score, logit, and probability ranges;
ensure loss consumes raw logits along the vocabulary axis;
compare initial loss with the uniform baseline $\ln8$ ;
compare final validation loss with the corpus floor, not zero.

4. Gradients and Updates

distinguish missing, zero, finite, and non-finite gradients;
record per-parameter and global norms before clipping;
verify clipping occurs before step();
compare update magnitude with parameter magnitude;
confirm the learning rate is nonzero and parameters actually change.

5. Experimental State

hold seeds and data fixed during ablations;
restore optimizer and RNG state for continuation;
record configuration and software environment;
rerun the one-sequence overfit test after structural changes.

Compact Failure Matrix

Observation	First discriminating test
initial loss near zero	decode targets and run future-token intervention
loss exactly constant	compare parameters before and after one step
loss oscillates or becomes non-finite	reduce rate and inspect first extreme value
one-sequence test fails	trace gradients and update order before adding data
training falls, validation does not	verify split, counts, and distribution
loss below the claimed population floor	recompute data frequencies and check leakage
reload logits differ	compare model config, vocabulary, weights, and mode
resumed next step differs	compare optimizer, scheduler, batch, and RNG state

Ablations Need Controls

For each change, state:

the hypothesis;
the one variable changed;
the variables held fixed;
the measurements collected;
the result;
the narrow conclusion;
what the result does not establish.

Changing initialization, optimizer, learning rate, corpus, and seed together is not an interpretable ablation.

Q1. Choose the first audit

A run begins with loss 0.02, far below both $\ln8$ and the corpus floor. What should you inspect before tuning the optimizer?

Choose the first audit

Inspect target alignment and future-token leakage before changing optimization.

Increase the learning rate because the initial loss is already small.

Add another decoder block before checking the data path.

Select one choice, then check.

Hint

The anomaly exists before the first update.

Solution

Decode input-target alignment and test the causal mask with a future-token intervention. Optimizer tuning cannot explain an impossible initial objective.

Generation Comes after the Training Audit

A trained checkpoint is ready for Chapter 8 only after its architecture, data, loss, gradients, validation evidence, and reload behavior pass these checks.