Exercises

Practice encoder and decoder masks, padding, corrupted inputs, shifted targets, leakage diagnosis, cross-attention sources and shapes, family selection, and specification audits.

Use explicit sequences, axes, and shift conventions. Do not answer from family names alone.

Q1. Construct encoder visibility

Write the $3\times3$ visibility matrix for a non-padded encoder input A B C under full self-attention.

Your answer

Answer it first, then check.

Hint

There is no causal or padding restriction.

Solution

[[1,1,1],[1,1,1],[1,1,1]]

Q2. Construct decoder visibility

Write the second row of an inclusive left-to-right causal visibility matrix for four positions.

Your answer

Compute it first, then check your number.

Hint

Include the diagonal under this convention.

Solution

The row is

[1,1,0,0]

Q3. Separate padding masks

In a length-five encoder record, positions 4 and 5 are padding. How many key columns may a real query read, and should padded query positions contribute to the task loss?

Your answer

Answer it first, then check.

Hint

Count content keys, then decide whether placeholder outputs are targets.

Solution

Three key columns remain readable, and padded query positions should be excluded from the task loss.

Q4. Align masked-token evidence

The clean input is red fox runs; the model input is red <mask> runs. Which token is the selected target, and may its query use runs under full encoder self-attention?

Your answer

Answer it first, then check.

Hint

Recover the changed token and distinguish corruption from attention masking.

Solution

The target is fox, and its query may use runs.

Q5. Build shifted decoder targets

For tokens A B C D, an input tensor contains A B C. Write its next-token target tensor.

Your answer

Answer it first, then check.

Hint

Drop the first token rather than the last.

Solution

The target tensor is B C D.

Q6. Detect decoder leakage

Changing future token D changes the logits used to predict B. Name the violated contract and the first two places to inspect.

Choose the contract and first checks

Causal dependence is violated; inspect the causal mask and input-target shift.

Only padding is violated; inspect the source padding mask.

Only weight tying is violated; inspect the unembedding matrix.

Select one choice, then check.

Hint

Inspect visibility and preprocessing.

Solution

The causal-dependence contract is violated. Inspect the causal mask and input-target shift first.

Q7. Assign cross-attention sources

In encoder-decoder cross-attention, which records supply $Q$ , and which supply $K$ and $V$ ?

Choose the Q/K/V sources

The decoder supplies Q; the encoder supplies K and V.

The encoder supplies Q, K, and V.

The decoder supplies Q, K, and V.

Select one choice, then check.

Hint

Rows correspond to target positions and columns to source positions.

Solution

The decoder supplies

Q

; the encoder supplies

K

and

V

Q8. Derive all attention score shapes

For $B=2$ , $h=4$ , $T_s=6$ , and $T_t=3$ , give encoder self-attention, decoder self-attention, and cross-attention score shapes.

Your answer

Answer it first, then check.

Hint

Use source-source, target-target, then target-source axes.

Solution

(2,4,6,6)

(2,4,3,3)

, and

(2,4,3,6)

Q9. Choose a family without overclaiming

A task needs one contextual output per input token, with both left and right context available. Name the natural starting family and state whether it is the only architecture that could be adapted.

Your answer

Answer it first, then check.

Hint

Separate a useful default from an impossibility claim.

Solution

Encoder-only is the natural starting family, but it is not the only architecture that could be adapted.

Q10. Audit a family specification

A document says only “T5-like Transformer.” List four additional facts needed to reconstruct its information flow and learning target.

Choose the four required facts

Source and target inputs; all masks; Q/K/V sources; target shift and included loss positions.

The T5-like label, parameter count, benchmark score, and model name.

Optimizer, learning rate, hardware, and checkpoint filename only.

Select one choice, then check.

Hint

Use the chapter's four defining questions and target ledger.

Solution

State source and target inputs, all visibility masks, Q/K/V sources for each attention operation, and target shift plus included loss positions. Readout and inference procedure should also be recorded.

Q1. Construct encoder visibility

Q2. Construct decoder visibility

Q3. Separate padding masks

Q4. Align masked-token evidence

Q5. Build shifted decoder targets

Q6. Detect decoder leakage

Q7. Assign cross-attention sources

Q8. Derive all attention score shapes

Q9. Choose a family without overclaiming

Q10. Audit a family specification

Pause and reflect