Exercises

Practice deriving self-attention shapes, calculating projections and masked weights, applying W_O, diagnosing axis and batching errors, counting parameters, and bounding interpretation claims.

These exercises move from shapes and individual calculations to debugging and evidence. Each prompt contains the values and conventions it needs.

Read and Derive Shapes

Q1. Complete the shape trace

A batch-major single-head layer uses $B=4$ , $T=12$ , $d_{model}=8$ , $d_k=3$ , and $d_v=5$ . What is the shape of the attention reading $H=AV$ ?

Choose one

(4, 12, 5)

(4, 12, 8)

(4, 12, 12)

(12, 3, 5)

Select one choice, then check.

Hint

For each batch item,

(T,T)(T,d_v)=(T,d_v)

Solution

H

has shape

(4,12,5)

. The output projection will later map its last axis from 5 to

d_{model}=8

Q2. Derive the output-projection shape

The same layer produces $H$ with shape $(4,12,5)$ and must return $Y$ with shape $(4,12,8)$ . What shape must $W_O$ have?

Choose one

(5, 8)

(8, 5)

(12, 8)

Select one choice, then check.

Hint

Complete

(4,12,5)(?,?)=(4,12,8)

Solution

W_O:(5,8)

Calculate the Operations

Q3. Project one query

Let $x=[2,-1]$ and

W_Q=\begin{bmatrix}1&2\\0&-1\end{bmatrix}.

What is the second coordinate of $q=xW_Q$ ?

Your answer

Compute it first, then check your number.

Hint

Multiply

x

by the second column of

W_Q

Solution

q=[2,5]

, so its second coordinate is 5.

Q4. Normalize an allowed causal row

A causal query has two allowed scores, $[\ln 3,0]$ , followed by one forbidden future score. What are the three attention weights after masking and softmax?

Choose the weight row

[0.75, 0.25, 0]

[0.5, 0.5, 0]

[0.6, 0.2, 0]

Select one choice, then check.

Hint

e^{\ln 3}=3

and

e^0=1

Solution

The allowed weights are

3/(3+1)=0.75

and

1/(3+1)=0.25

. The forbidden future weight is 0.

Q5. Finish the value and output calculation

Weights are $[0.75,0.25]$ , values are $v_1=[2,0]$ and $v_2=[0,4]$ , and

W_O=\begin{bmatrix}1&1\\0&2\end{bmatrix}.

What is the second coordinate of $y=(0.75v_1+0.25v_2)W_O$ ?

Your answer

Compute it first, then check your number.

Hint

Compute

h

first, then multiply it by the second column of

W_O

Solution

h=0.75[2,0]+0.25[0,4]=[1.5,1]

. Its second projected coordinate is

1.5(1)+1(2)=3.5

Diagnose and Repair

Q6. Find the wrong axis

An attention matrix is

A=\begin{bmatrix}0.8&0.3\\0.2&0.7\end{bmatrix}.

Its columns sum to 1, but its rows do not. Which implementation error best explains the matrix?

Choose one

Softmax was applied down the query axis instead of across key positions

The scores were divided by sqrt(d_k)

The value projection has width d_v

Select one choice, then check.

Hint

Each query needs one distribution over keys.

Solution

Softmax used the wrong axis. It must normalize each row across its key-position columns.

Q7. Detect cross-batch mixing

A batch contains sequences A and B. After changing only one token in B, the output for A also changes. All model parameters are fixed and there is no batch-dependent normalization. What does this observation establish?

Choose one

The implementation is mixing or coupling batch items somewhere

This is expected because projection parameters are shared

The causal mask is correctly blocking future positions

Select one choice, then check.

Hint

A deterministic per-sequence operation must preserve independence between batch items.

Solution

The batch dimension was likely transposed, flattened, normalized, or multiplied incorrectly. Changing B must not alter A under the stated layer.

Count and Interpret Carefully

Q8. Count projection parameters

A bias-free single head has $d_{model}=8$ , $d_k=3$ , and $d_v=4$ . Count all entries in $W_Q$ , $W_K$ , $W_V$ , and $W_O$ .

Your answer

Compute it first, then check your number.

Hint

Count each matrix separately, including the map from

d_v

back to

d_{model}

Solution

24+24+32+32=112

parameters.

Q9. Bound an interpretation claim

For one prediction, position 7 receives the largest attention weight in one head. Which conclusion is justified by that observation alone?

Choose the supported claim

Its value received the largest scalar multiplier in that row and head

Position 7 caused the final model prediction

The model's reasoning is located at position 7

Select one choice, then check.

Hint

Values, output projections, residual paths, and later layers have not been examined.

Solution

The supported claim is only that the value at position 7 received the largest scalar multiplier in that attention row and head. Causal or reasoning claims require additional evidence.

Integrated Audit

Q10. Specify a complete layer audit

A colleague reports only the final output shape of a new causal self-attention implementation. Choose the smallest audit below that checks the chapter's main failure modes.

Choose the audit

Check every intermediate shape, score scale, mask before softmax, finite row sums, forbidden zeroes, output width, selected numerical rows, and cross-batch independence

Check only that the final output has shape (B, T, d_model)

Plot one attention heat map and decide whether the pattern looks plausible

Select one choice, then check.

Hint

Prefer checks that can locate the first incorrect intermediate.

Solution

The complete audit is required. It checks the computation in dependency order and separates observed tensors from interpretation.

Read and Derive Shapes

Q1. Complete the shape trace

Q2. Derive the output-projection shape

Calculate the Operations

Q3. Project one query

Q4. Normalize an allowed causal row

Q5. Finish the value and output calculation

Diagnose and Repair

Q6. Find the wrong axis

Q7. Detect cross-batch mixing

Count and Interpret Carefully

Q8. Count projection parameters

Q9. Bound an interpretation claim

Integrated Audit

Q10. Specify a complete layer audit

Pause and reflect