Audit Masks, Targets, Vocabulary, and Readout

Audit range checks, masks at every block, target shift, normalization and softmax axes, tied identity, and valid loss denominators. Use failure signatures and interventions to find valid-shape errors.

End-to-end tensor shapes can remain valid while a decoder reads future tokens, predicts the current token, normalizes along time, or transposes a tied matrix incorrectly. Audit the model's information and index contracts, not only its ability to return logits.

Boundary Checks

Reject invalid states early:

token IDs outside $[0,V)$ ;
sequence lengths outside $[1,C]$ ;
target IDs outside $[0,V)$ ;
input and target shapes that do not match;
a valid-target mask with no included positions.

Silently clipping an ID or reusing the last position row changes the learning problem and conceals the source of the error.

Information Checks

Change one later input token while holding parameters and earlier tokens fixed. All earlier logits must remain unchanged. Repeat this intervention at several positions and in every block if intermediate access is available.

A triangular mask displayed in logs is not enough. Confirm it is applied before softmax, broadcast to the intended batch and head axes, and used by every causal attention layer.

Target and Readout Checks

For input [0,1,2,3], targets must be [1,2,3,7] under the frozen example. Verify that:

logits row $t$ is paired with target row $t$ after data shifting;
no second hidden shift occurs inside the loss;
softmax or log-sum-exp reduces the vocabulary axis;
final LayerNorm reduces the model-feature axis;
tied logits use $E^\top$ , not $E$ under incompatible axes;
padded or ignored targets are absent from the mean denominator.

Failure Signatures

Observation	First contract to inspect
loss is near zero before learning	current-token target or future leakage
earlier logits change after editing a future token	mask construction or application
probability rows do not sum to 1	softmax axis or numerical stability
adding batch items changes an existing example	batch-axis mixing
tied model reports 400 parameters	unembedding may still be independent
output has width 4 instead of 8	unembedding missing or transposed
position 5 runs despite $C=4$	position-range validation missing

Modify one entry of $E$ in a controlled copy. The corresponding token lookup and vocabulary logit calculation must both change. Restore the entry after the test. This detects copied equality masquerading as shared identity.

Q1. Diagnose a suspicious tied count

The frozen model is configured as tied but reports 400 learned entries. What is the most likely extra parameter?

Choose the likely extra parameter

An independent 4-by-8 unembedding matrix.

One extra four-value position row.

An eight-value readout bias.

Select one choice, then check.

Hint

Compare the tied total 368 with the untied total 400.

Solution

An independent

(4,8)

unembedding matrix adds the unexpected 32 entries.

Training Starts Only After These Checks

An optimizer can reduce the wrong objective. Chapter 7 begins with the frozen architecture and repeats the causality, target, shape, and parameter tests before interpreting any falling loss as learning.