Review

Training is an experiment around a fixed model. The model maps input IDs to logits; the data defines targets; the loss compares them; backpropagation calculates gradients; the optimizer changes parameters; held-out data tests the result.

Reconstruct the Data Contract

For any five-token sequence $[s_0,s_1,s_2,s_3,s_4]$:

A batch has IDs and targets $(B,4)$, hidden records $(B,4,4)$, and logits $(B,4,8)$. Cross-entropy receives logits $(4B,8)$ and target IDs $(4B)$ after a matched reshape.

Decode sample rows before training. Shape checks cannot detect a wrong shift or a sequence-boundary leak.

Reconstruct the Model Contract

The architecture retains:

• 32 token-embedding entries;
• 16 position entries;
• two 156-entry pre-norm decoder blocks;
• 8 final-LayerNorm entries;
• one shared embedding/readout table.

The total is 368. readout.weight is token_embedding.weight tests sharing; equal initial values do not.

Reconstruct One Step

train mode
-> clear old gradients
-> forward logits
-> cross-entropy with shifted targets
-> backward
-> inspect and optionally clip gradients
-> optimizer update
-> log selected evidence

Clipping after the update is too late. Softmax before cross-entropy is unnecessary and less stable. Missing zero_grad unintentionally accumulates gradients across steps.

Reconstruct the Evidence Ladder

1. Static contracts: IDs, targets, shapes, parameter count, tied identity.
2. Information flow: future-token interventions leave earlier logits fixed.
3. One-step behavior: finite loss and gradients; parameters change.
4. One-sequence overfit: loss can approach zero on one deterministic row.
5. Corpus training: loss approaches the uncertainty supported by the data.
6. Validation: fresh same-distribution sequences behave similarly.
7. Reload and resume: stored state reproduces logits and the next update in the recorded environment.

Each level answers a different question. No single loss curve replaces them.

Reconstruct the Corpus Floor

After <bos>, probabilities $(1/2,1/4,1/4)$ contribute $(3/2)\ln2$. After <bos> A B, the C/D choice contributes $(1/2)\ln2$ after weighting by how often that prefix occurs. Divide by four target positions:

The verified validation result 0.3579 is close to this floor. The one-sequence loss 0.0019 answers a different question because that dataset has no ambiguous target.

Reconstruct Reproducibility State

Weights reproduce current inference only when model definition, configuration, vocabulary, and mode also match. Exact continuation additionally needs optimizer state, step, batch-generator state, scheduler state if present, data identity, and environment details.

Seeds reduce variation within a controlled environment. They do not guarantee identical results across PyTorch releases, platforms, or devices.

Audit before Generating

When a result is suspicious, inspect data and information flow before tuning the optimizer. Then inspect forward values, loss, gradients, updates, validation, and checkpoint state in that order.

Chapter 8 begins with a trained and audited checkpoint. It will separate prompt prefill from repeated one-token decoding and verify cached against uncached logits.