Exercises

Practice target construction, leakage detection, shapes, parameter counting, entropy, update order, gradient clipping, train-validation diagnosis, checkpoint state, and end-to-end failure audits.

Use vocabulary IDs <bos>=0, A=1, B=2, C=3, D=4, X=5, Y=6, and <eos>=7. The model has $V=8$ , $C=4$ , $d_{model}=4$ , two blocks, two heads, $d_{ff}=8$ , and tied input-output weights.

Q1. Construct an aligned example

For sequence <bos> A B D <eos>, write the input-ID row and target-ID row.

Your answer

Answer it first, then check.

Hint

Remove the final token for input and the first token for target.

Solution

Input [0,1,2,4]; target [1,2,4,7].

Q2. Identify target leakage

A training row uses input [0,1,2,3,7] and calculates loss only for target 7 at its final input position. Which judgment is correct?

Choose one

The target token is already present at the position used to predict it.

The causal mask always removes the token at the current position.

The only problem is that the vocabulary has the wrong size.

Select one choice, then check.

Hint

Causal attention hides later positions, not the query position itself.

Solution

This leaks the target. To predict <eos>, the final input position should contain C, not <eos>.

Q3. Audit batch and loss shapes

For $B=5$ and $T=4$ , give the shapes of input IDs, logits, flattened logits, and flattened targets.

Your answer

Answer it first, then check.

Hint

BT=20

and

V=8

Solution

(5,4)

(5,4,8)

(20,8)

, and

(20)

Q4. Recover the parameter ledger

The token table has 32 entries, positions 16, each block 156, and final LayerNorm 8. Calculate the total for two blocks with tied readout.

Your answer

Compute it first, then check your number.

Hint

Calculate

32+16+2(156)+8

Solution

32+16+312+8=368

Q5. Calculate irreducible loss

After <bos>, the target entropy is $(3/2)\ln2$ . The ambiguous C/D context contributes another $(1/2)\ln2$ per sequence on average. There are four target positions. Calculate the population loss floor in nats per token.

Your answer

Compute it first, then check your number.

Hint

Add the contributions before dividing by four.

Solution

[(3/2)\ln2+(1/2)\ln2]/4=\ln2/2\approx0.346574

Q6. Repair a training step

The following order is wrong: forward, backward, optimizer step, clip gradients, clear gradients. Choose the corrected order.

Choose one

Clear gradients, forward and loss, backward, clip, optimizer step.

Forward, optimizer step, backward, clear gradients, clip.

Clear gradients, forward, optimizer step, clip; backward is optional.

Select one choice, then check.

Hint

The optimizer step consumes gradients created by backward.

Solution

Clear old gradients; run forward and loss; run backward; inspect or clip current gradients; then update.

Q7. Interpret clipping evidence

Before clipping, the global gradient norm is 3.0. After clipping at threshold 1.0, it is 1.0. What does this observation establish?

Choose one

This step's global gradient magnitude was bounded; the data and objective still require separate checks.

The causal mask and targets must be correct because clipping succeeded.

A threshold of 1.0 is optimal for every Transformer.

Select one choice, then check.

Hint

Clipping does not inspect target alignment.

Solution

The update uses a bounded gradient magnitude on that step. Nothing in this measurement proves correct targets, causality, or a universal threshold.

Q8. Distinguish two failure patterns

Run A stays near 2.05 on both training and validation. Run B falls to 0.02 on training but rises to 3.0 on validation. Match each run to the first broad diagnosis.

Choose the diagnoses

A suggests an optimization or update failure; B suggests severe overfitting or split mismatch.

Both runs primarily show successful generalization.

A shows severe overfitting; B shows that no useful update occurred.

Select one choice, then check.

Hint

Ask whether training itself improves before comparing it with validation.

Solution

A first suggests that useful updates are not occurring. B suggests a large train-validation gap, caused by memorization, distribution mismatch, or split leakage that must be inspected.

Q9. Choose checkpoint contents

Which record is sufficient for an exact next AdamW update in the same environment?

Choose one

Model weights only.

Model weights and the original model seed only.

Model and optimizer state, step, configuration, data identity, and relevant RNG state.

Select one choice, then check.

Hint

AdamW stores moving averages, and sampling determines the next examples.

Solution

The complete record is required. Add scheduler state too if a scheduler is used.

Q10. Diagnose a suspicious run

A model reports correct shapes and finite loss. Its initial loss is 0.03, its tied parameter count is 400, and changing input position 3 changes logits row 0. Identify the three independent failures or suspicions.

Choose the three issues

Target or future leakage; an independent unembedding; and causal leakage.

A non-finite loss; a wrong logits shape; and missing gradients.

Only a batch-seed mismatch and ordinary floating-point noise.

Select one choice, then check.

Hint

Compare loss with ln 8, parameters with 368, and the intervention with causality.

Solution

The near-zero initial loss suggests target or future leakage; 400 entries indicate an independent readout instead of true tying; and the future intervention directly proves causal leakage.

Q1. Construct an aligned example

Q2. Identify target leakage

Q3. Audit batch and loss shapes

Q4. Recover the parameter ledger

Q5. Calculate irreducible loss

Q6. Repair a training step

Q7. Interpret clipping evidence

Q8. Distinguish two failure patterns

Q9. Choose checkpoint contents

Q10. Diagnose a suspicious run

Pause and reflect