Exercises

Practice ID validation, lookup, positions, block parameters, shapes, tied logits, shared identity, cross-entropy, masked means, causal interventions, and end-to-end failure diagnosis.

Use the frozen vocabulary, dimensions, position table, and parameter conventions. Show intermediate values for numerical answers.

Q1. Validate an ID window

Is input [0,2,7,8] valid for the frozen model? If not, identify the first invalid value.

Your answer

Answer it first, then check.

Hint

Compare each ID with

V=8

Solution

No. ID 8 is outside

[0,8)

Q2. Calculate position-aware lookup

Token B has row $[0,0,1,0]$ . What record represents it at position 2?

Your answer

Compute it first, then check your number.

Hint

Use the third row of the frozen position table.

Solution

[0,0,1,0]+[0,0,0.1,0]=[0,0,1.1,0]

Q3. Count one decoder block

Under the frozen conventions, count attention matrices, biased MLP, and two LayerNorms for one block.

Your answer

Compute it first, then check your number.

Hint

Do not add attention biases; do include MLP biases.

Solution

4(4\times4)+(4\times8+8+8\times4+4)+2(8)=156

Q4. Track every major shape

For $B=2,T=3$ , give the ID, residual-stream, final-record, and logits shapes.

Your answer

Answer it first, then check.

Hint

Use ID rank 2, model width 4, and vocabulary width 8.

Solution

(2,3)

(2,3,4)

(2,3,4)

, and

(2,3,8)

Q5. Calculate two tied logits

For $h=[1,-2,3,0.5]$ , calculate tied logits for D with embedding $[1,1,0,0]$ and Y with embedding $[0,0,1,1]$ .

Your answer

Answer it first, then check.

Hint

Add coordinates selected by the 1s in each embedding.

Solution

z_D=1-2=-1

and

z_Y=3+0.5=3.5

Q6. Distinguish sharing from copying

Two matrices begin numerically equal, but changing an embedding entry leaves the output matrix unchanged. Are the weights tied?

Your answer

Answer it first, then check.

Hint

Ask whether one update affects both operations.

Solution

No. The implementation copied values into independent parameters.

Q7. Calculate one cross-entropy

For logits $[0,0]$ and target index 1, calculate natural-log cross-entropy.

Your answer

Compute it first, then check your number.

Hint

Use

-\log(1/2)

Solution

-\log(0.5)=\log2\approx0.693147

Q8. Calculate a masked mean loss

Per-position losses are $[1,2,9,9]$ and valid-target mask is $[1,1,0,0]$ . What mean loss should be reported?

Your answer

Compute it first, then check your number.

Hint

Do not divide by all four positions.

Solution

(1+2)/(1+1)=1.5

Q9. Design a causal intervention

Using zero-based indices, to test whether logits row 1 reads future position 3, what should you change and what should remain invariant?

Choose the intervention and invariant

Change only token 3; logits row 1 must remain unchanged.

Change token 0; logits row 1 must remain unchanged.

Change model weights; every logits row must remain unchanged.

Select one choice, then check.

Hint

Alter only information the query is forbidden to read.

Solution

Change only the token at position 3. With fixed parameters and earlier inputs, logits row 1 must remain unchanged.

Q10. Audit a misleading successful run

A model returns shape $(2,4,8)$ and finite loss, but reports 400 parameters while configured as tied. Changing the token at the fourth input position (index 3) also changes logits row 1. Identify both independent failures.

Choose both failures

An independent unembedding parameter and causal leakage.

A logits-shape error and a non-finite loss.

A missing position row and an invalid vocabulary ID.

Select one choice, then check.

Hint

Compare parameter count with 368 and test forbidden dependence.

Solution

The model likely stores an independent unembedding matrix, adding 32 entries, and its causal mask or preprocessing permits future leakage.

Q1. Validate an ID window

Q2. Calculate position-aware lookup

Q3. Count one decoder block

Q4. Track every major shape

Q5. Calculate two tied logits

Q6. Distinguish sharing from copying

Q7. Calculate one cross-entropy

Q8. Calculate a masked mean loss

Q9. Design a causal intervention

Q10. Audit a misleading successful run

Pause and reflect