Overfit One Sequence before Training the Corpus
Use one deterministic input-target row as the smallest end-to-end training diagnostic. Interpret near-zero loss alongside independent parameter, gradient, and causality checks.
The smallest complete training test uses one sequence repeatedly. The objective is not generalization. It is to answer a narrower question: can this model, loss, backward pass, and optimizer reduce error on one fixed example?
Use:
input: <bos> A B C
target: A B C <eos>
The four targets are deterministic in this one-example dataset. A model with sufficient capacity can therefore drive its training loss close to zero.
What a Successful Overfit Test Establishes
Under the frozen CPU experiment, 500 AdamW updates at learning rate 0.01 reduce the one-sequence loss from 2.1665 to 0.0019. This is strong evidence that:
- the registered parameters can receive gradients;
- the optimizer changes them;
- the loss is connected to the logits;
- the model can represent this one input-target mapping.
It does not establish that the causal mask is correct, that validation will improve, or that the model learned the four-pattern corpus rule. A model can memorize one sequence through an invalid future-token path.
Keep Architecture Audits beside the Test
Before accepting the falling loss:
- verify 368 learned entries and tied identity;
- verify input and target rows by decoding them;
- perturb the final input token and test invariance of earlier logits;
- confirm every parameter intended to learn has a finite gradient;
- confirm at least one parameter changes after
optimizer.step().
Failure Narrows the Search
| Observation | Inspect first |
|---|---|
| loss is exactly unchanged | zero learning rate, missing step, detached loss, or no gradients |
| loss becomes non-finite | logits, loss stability, gradients, or update scale |
| only readout changes | registration or blocked earlier gradients |
| loss falls but causal test fails | mask construction or application |
| one sequence fits but corpus does not | data rule, capacity, optimizer, or ambiguous targets |
The test is useful because its demand is intentionally easy. If the model cannot fit one example, increasing corpus size makes diagnosis harder.
Q1. State the limit of the overfit test
A one-sequence loss reaches 0.001, but changing its final input token changes logits row 0. Is the model ready for corpus training?
Select one choice, then check.
Hint
Solution
Increase Difficulty One Dependency at a Time
After this test passes, restore all four corpus patterns and an independent validation collection. Do not change architecture, initialization, optimizer, and data at the same time; otherwise a new failure has no clear source.