Exercises

Practice residual traces, normalization statistics, norm ordering, MLP shapes, position independence, GELU, gated parameter counts, dropout modes, and silent block failures.

Use the stated convention in each problem. Show intermediate values when a problem asks for a calculation.

Q1. Trace both residual additions

For one token, $x=[1,2]$ , attention returns $[-0.5,1]$ , and the MLP returns $[2,-1]$ . Calculate $u$ and $y$ .

Your answer

Answer it first, then check.

Hint

First use

u=x+a

, then

y=u+m

Solution

u=[0.5,3]

and

y=[2.5,2]

Q2. Calculate LayerNorm statistics

Ignoring epsilon, calculate the mean and population variance of $[2,4,6]$ .

Your answer

Answer it first, then check.

Hint

Divide the squared-deviation sum by 3.

Solution

The mean is

4

and the variance is

(4+0+4)/3=8/3

Q3. Separate LayerNorm from RMSNorm

Which operation is unchanged before learned affine parameters when the same constant is added to every coordinate: LayerNorm or RMSNorm?

Your answer

Answer it first, then check.

Hint

Only one operation subtracts a mean.

Solution

LayerNorm. RMSNorm retains the shifted direction.

Q4. Order a pre-norm block

Put these operations in order: MLP normalization, attention residual addition, MLP residual addition, attention normalization.

Choose the correct order

attention normalization, attention residual addition, MLP normalization, MLP residual addition

attention residual addition, attention normalization, MLP residual addition, MLP normalization

attention normalization, MLP normalization, attention residual addition, MLP residual addition

Select one choice, then check.

Hint

Use

u=x+A(N_1(x))

before

y=u+M(N_2(u))

Solution

Attention normalization; attention residual addition; MLP normalization; MLP residual addition.

Q5. Derive MLP shapes

For $d_{model}=6$ and $d_{ff}=24$ , give the shapes of $W_{up}$ and $W_{down}$ .

Your answer

Answer it first, then check.

Hint

Follow model width to hidden width and back.

Solution

W_{up}:(6,24)

and

W_{down}:(24,6)

Q6. Test position-wise independence

An MLP receives four token records. You alter only record 3. Which output records may change in a correct position-wise implementation?

Your answer

Answer it first, then check.

Hint

The MLP is applied independently at each position.

Solution

Only output record 3 may change.

Q7. Calculate a GELU value

Using $\operatorname{GELU}(x)=x\Phi(x)$ and $\Phi(1)\approx0.841345$ , calculate GELU $(1)$ .

Your answer

Compute it first, then check your number.

Hint

Substitute

x=1

directly.

Solution

1\times0.841345=0.841345

Q8. Count a gated MLP

Ignoring all biases, a gated MLP has two $(d_{model},d_{ff})$ input projections and one $(d_{ff},d_{model})$ output projection. How many parameters does it have for $d_{model}=4$ , $d_{ff}=6$ ?

Your answer

Compute it first, then check your number.

Hint

Count both gate/value projections and the down projection.

Solution

2(4)(6)+(6)(4)=72

Q9. Compare dropout modes

With inverted dropout, $p=0.5$ , $z=6$ , and a retained mask entry, give the training value and evaluation value.

Your answer

Answer it first, then check.

Hint

Training divides by

1-p

Solution

Training returns

6/0.5=12

; evaluation returns

6

Q10. Design a block audit

A block has the correct final shape but changing token 2 changes token 1 during the MLP-only stage. Name the violated contract and the first implementation axis or operation to inspect.

Choose the contract and first inspection

Position-wise independence is violated; inspect whether an MLP reshape, reduction, or projection mixes the sequence axis.

Causality is violated; inspect only the attention mask.

Vocabulary alignment is violated; inspect token IDs only.

Select one choice, then check.

Hint

Ask which axis an MLP is allowed to mix.

Solution

The position-wise independence contract is violated. Inspect whether an MLP projection, reshape, or reduction treats the sequence axis as a feature axis.

Q1. Trace both residual additions

Q2. Calculate LayerNorm statistics

Q3. Separate LayerNorm from RMSNorm

Q4. Order a pre-norm block

Q5. Derive MLP shapes

Q6. Test position-wise independence

Q7. Calculate a GELU value

Q8. Count a gated MLP

Q9. Compare dropout modes

Q10. Design a block audit

Pause and reflect