Audit a Tiny Pre-Norm Block
An executable tiny block exposes normalization, both residual updates, and the position-wise MLP. Shape checks, independence tests, and branch ablations locate errors that a plausible final tensor can hide.
A full block can return the expected final shape while hiding a wrong normalization axis, a position-mixing MLP, or an omitted residual update. A useful audit exposes intermediate values and tests one contract at a time.
Build the Smallest Honest Trace
The example below uses two token records, feature-wise LayerNorm, a supplied attention update, and a small ReLU MLP. Supplying the attention update isolates the block mechanics already studied here; earlier chapters audit how attention itself is calculated.
Trace a pre-norm block
Run the block, inspect both residual additions, then alter one token before the position-wise MLP test.
Ready to run.
Use Branch Ablations as Diagnostics
Run four related cases without retraining:
- the complete block;
- attention update fixed to zero;
- MLP update fixed to zero;
- both updates fixed to zero.
The fourth case must return exactly when no other operation sits after the residual additions. The first three cases reveal where a discrepancy first appears. They do not measure the quality of separately trained architectures.
Failure Signatures
| Observation | First contract to inspect |
|---|---|
| Changing token 2 changes token 1 inside the MLP | position-wise independence |
| Normalized values depend on another batch item | normalization axis |
| Zeroing both branches does not return the input | residual path or misplaced normalization |
| Training output changes but evaluation output also changes | dropout mode or another random operation |
| Final width differs from model width | down projection or branch shape |
Q1. Choose an isolation test
You suspect that an MLP accidentally mixes sequence positions. Which single input change gives the clearest test?
Select one choice, then check.
Hint
Solution
Carry the Contract Forward
Later architectures will change masks, attention connections, and sometimes normalization or MLP variants. Continue to record the exact block equations, intermediate shapes, mode, and ablations before attributing behavior to a model name.