Exercises

Practice permutation equivariance, absolute-vector addition, sinusoidal values, signed relative offsets, clipping, RoPE rotations and offsets, ALiBi softmax, and valid-shape position audits.

These exercises test reordering, absolute and relative position calculations, RoPE geometry, ALiBi scores, and position audits. Each prompt states its index and sign conventions.

Reordering and Absolute Position

Q1. Name the reordering property

An unmasked content-only self-attention layer satisfies $f(PX)=Pf(X)$ for every row permutation matrix $P$ . Which property does this equation state?

Choose one

Permutation equivariance

Permutation invariance

Causal masking

Layer normalization

Select one choice, then check.

Hint

An invariant output would not contain the leading

P

Solution

The equation states permutation equivariance.

Q2. Reorder an unmasked output

An unmasked content-only layer returns rows $f(X)=[a,b,c]$ . The input rows are reordered from $[0,1,2]$ to $[2,0,1]$ . What output row order follows from permutation equivariance?

Choose one

[c, a, b]

[a, b, c]

[c, b, a]

Select one choice, then check.

Hint

Select output rows 2, 0, and 1 in that order.

Solution

The output is

[c,a,b]

Q3. Add an absolute position vector

A repeated token has embedding $e=[1,2,0]$ . At index 4 its learned position vector is $p_4=[-0.5,0.25,1]$ . What is the second coordinate of $e+p_4$ ?

Your answer

Compute it first, then check your number.

Hint

Add corresponding coordinates.

Solution

e+p_4=[0.5,2.25,1]

, so the requested coordinate is

2.25

Q4. Calculate one sinusoidal value

For $d_{model}=4$ , the first sine coordinate is $PE(pos,0)=\sin(pos)$ . What is $PE(0,0)$ ?

Your answer

Compute it first, then check your number.

Hint

The position is the angle for the first pair.

Solution

PE(0,0)=\sin(0)=0

Relative Offsets and RoPE

Q5. Apply the offset convention

Relative offset is defined as key index minus query index, $j-i$ . A query at index 8 reads a key at index 5. Which offset is used?

Your answer

Compute it first, then check your number.

Hint

Substitute

j=5

and

i=8

Solution

j-i=5-8=-3

Q6. Clip a relative offset

A relation table clips $j-i$ to $[-4,4]$ . The original offset is $-9$ . Which table offset is selected?

Your answer

Compute it first, then check your number.

Hint

Clamp the value to the nearest boundary.

Solution

The selected offset is

-4

Q7. Rotate a coordinate pair

Using $R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix}$ , rotate $[0,2]^T$ by $\phi=\pi/2$ . What is the first coordinate?

Your answer

Compute it first, then check your number.

Hint

\pi/2

, cosine is 0 and sine is 1.

Solution

The rotated vector is

[-2,0]^T

, so the first coordinate is

-2

Q8. Compare equal RoPE offsets

The same unrotated query and key pair uses the same RoPE frequency. Which two position pairs must produce the same pairwise RoPE dot product?

Choose one

(query 1, key 4) and (query 7, key 10)

(query 1, key 4) and (query 7, key 9)

(query 1, key 4) and (query 1, key 5)

Select one choice, then check.

Hint

The identity depends on key position minus query position.

Solution

The pairs

(1,4)

and

(7,10)

both have offset 3.

Score Bias and Integrated Audit

Q9. Normalize an ALiBi row

A causal query has four equal content scores. ALiBi adds biases $[-1.5,-1.0,-0.5,0]$ . What is the softmax weight on the final key? Give at least three decimal places.

Your answer

Compute it first, then check your number.

Hint

Use stable softmax; subtracting the maximum 0 leaves the same four values.

Solution

The denominator is

e^{-1.5}+e^{-1}+e^{-0.5}+1\approx2.1975

, so the final weight is

1/2.1975\approx0.455

Q10. Audit a valid-shape position mismatch

Training uses left padding and assigns the first real token position 0. Evaluation also uses left padding but assigns physical column indices, so the first real token may receive position 3. All input, mask, and output shapes match. Which audit is most direct?

Choose the audit

Print token, padding, and position-ID mappings for matched training and evaluation examples

Check the position-ID tensor shape again

Increase d_model until outputs match

Normalize attention across query positions

Select one choice, then check.

Hint

Inspect values at the boundary where tokens acquire position IDs.

Solution

Print and compare the token-to-position mapping. Training and evaluation must use the same declared padding and index policy.

Reordering and Absolute Position

Q1. Name the reordering property

Q2. Reorder an unmasked output

Q3. Add an absolute position vector

Q4. Calculate one sinusoidal value

Relative Offsets and RoPE

Q5. Apply the offset convention

Q6. Clip a relative offset

Q7. Rotate a coordinate pair

Q8. Compare equal RoPE offsets

Score Bias and Integrated Audit

Q9. Normalize an ALiBi row

Q10. Audit a valid-shape position mismatch

Pause and reflect