Milestone 4 of 7

Train and evaluate the corpus

Run the frozen configuration, compare losses with known baselines, inspect fixed prompts, and verify checkpoint reload and resume.

Run the frozen configuration on the sampled training corpus while keeping the validation rows outside parameter updates and tuning decisions.

Milestone goal

Produce a training record, an independent validation record, fixed-prompt probabilities, and a checkpoint that passes both inference reload and exact next-step resume tests.

Preserve the sampled data

Generate 256 training sequences with seed 17 and 1,024 validation sequences with seed 29. Save the four realized pattern counts with the checkpoint. Equal population probabilities do not imply exactly equal finite counts.

Draw each training batch with its own recorded generator state. Validation always evaluates the full frozen validation tensor in evaluation mode and without gradients. Do not regenerate the validation sample every time its loss is measured.

Record more than the batch loss

At steps 0, 1, 2, 5, 10, 25, 50, 100, 250, 500, and 1,000, record:

sampled-batch loss before the update;
complete training-set loss after the update;
complete validation-set loss after the update;
global gradient norm before and after clipping;
learning rate;
probabilities for fixed contexts such as <bos>, <bos> A B, and <bos> X Y.

The batch loss is noisy evidence about one update. The complete train and validation losses support broader statements about the frozen samples.

Interpret the two reference levels

$\log 8\approx2.0794$ is the uniform eight-token baseline.
$\log 2/2\approx0.3466$ is the population conditional-entropy floor under the toy distribution.

The checked run finishes near 0.34943 training loss and 0.35794 validation loss. These values are close to the population floor and far below uniform, which shows that the model captured the toy conditional structure. They do not show broad language ability, and the finite validation value need not equal the population floor exactly.

Inspect the conditional probabilities rather than reporting loss alone. After <bos> A B, probability should be shared between C and D; after <bos> X Y, X should dominate. A model that always chooses one A-B ending can have plausible samples while misrepresenting the learned distribution.

Save enough state for two distinct promises

An inference checkpoint needs model weights and the configuration required to reconstruct the model. Exact training continuation also needs:

optimizer state;
completed step;
batch-generator state;
all other random states that affect the next update;
vocabulary, data/split identity, and realized counts;
dtype, device, framework version, and update settings.

Test the promises separately:

Reload: construct a fresh model, load its state, and require bitwise- equal probe logits in the checked CPU environment.
Resume: restore model, optimizer, and batch-generator state, take one update in both uninterrupted and restored runs, and require equal resulting parameter tensors.

If inference reload passes but resume fails, first inspect optimizer moments, the next sampled batch, and omitted random state.

Question. Diagnose diverging evidence

Training loss falls while validation loss rises. Which action is justified?

Choose one

Preserve the run and inspect overfitting, the split, and configuration before claiming improvement.

Declare generalization because optimization succeeded.

Delete validation because it disagrees with training.

Select one choice, then check.

Hint

A lower training loss is not held-out performance.

Solution

Preserve the run and inspect overfitting, split integrity, and the frozen configuration. Do not claim improved generalization.

Diagnose the run in dependency order

Observation	First check
train and validation start unusually low	leakage or incorrect targets
train falls but validation rises	overfitting and split distribution
both plateau near uniform	update path, capacity, and learning rate
loss beats the theoretical floor substantially	sample imbalance, memorization, or metric mismatch
reload differs	model reconstruction, mode, dtype, and state keys
next-step resume differs	optimizer, batch generator, and random state

Acceptance gate

Continue only when:

the run uses the frozen configuration and preserved sampled rows;
training and validation evidence is logged at fixed steps;
fixed-context probabilities agree with the corpus conditionals;
inference reload and exact next-step resume both pass;
every claim names whether it concerns the batch, sampled split, population, or broader language behavior.

Deliverable: raw metric history, plotted train/validation traces, fixed-prompt probabilities, serialized checkpoint, reload and resume results, and exact rerun command.