Exercises

Practice greedy decoding, temperature, prefill and decode, cache shapes, attention scores, positions, numerical tolerance, memory, batched stopping, and cache-failure diagnosis.

Use the frozen model with $V=8$ , $C=4$ , $L=2$ , $h=2$ , and $d_h=2$ unless an exercise gives other values.

Q1. Trace a greedy generation step

Prompt IDs are [0,1]. The final logits row is [0,1,4,2,3,-1,0,0]. Which ID is appended by greedy decoding, and what is the extended input?

Your answer

Answer it first, then check.

Hint

Greedy decoding uses argmax.

Solution

Append ID 2, giving [0,1,2].

Q2. Apply temperature

Scale logits [2,0,-2] with temperature 0.5.

Your answer

Compute it first, then check your number.

Hint

Temperature acts before softmax.

Solution

The scaled row is [4,0,-4].

Q3. Distinguish prefill and decode

Which operation belongs to one-token decode after a prompt has already filled the cache?

Choose one

Project one new Q/K/V row, append K/V, and calculate one new logits row.

Recompute every earlier token's MLP and K/V records.

Backpropagate through the prompt and update parameters.

Select one choice, then check.

Hint

The prompt is already represented in the cache.

Solution

Decode projects the current token, appends its K/V per layer, and calculates its logits row against the cached prefix.

Q4. Calculate cache shape and count

For the frozen model at $B=1$ and $T=3$ , give each layer's K tensor shape and the total K/V scalar count across both layers.

Your answer

Answer it first, then check.

Hint

Use

2(2)(1)(2)(3)(2)

for total K and V.

Solution

Each K has shape

(1,2,3,2)

; all K/V tensors contain 48 scalars.

Q5. Calculate one attention score

For $d_h=2$ , query $[2,1]$ , and key $[1,-1]$ , calculate the scaled dot-product score.

Your answer

Compute it first, then check your number.

Hint

2(1)+1(-1)=1

Solution

1/\sqrt2\approx0.707107

Q6. Continue the position index

A prompt of length 2 is cached. Two generated tokens have since been consumed as inputs. What position index would the next input require, and is it valid for $C=4$ ?

Your answer

Answer it first, then check.

Hint

Add consumed prompt and generated-input counts.

Solution

The next index is 4, which is outside the valid range

[0,4)

Q7. Interpret logit error

Cached and full logits differ by $3\times10^{-6}$ under tolerance $10^{-5}$ for one float32 run. What is the appropriate conclusion?

Choose one

This comparison passes the recorded tolerance; other prefixes and intermediates still need checks.

The implementations are mathematically proven identical for all devices.

Any nonzero floating-point difference proves a cache error.

Select one choice, then check.

Hint

Compare the measured error with the acceptance threshold.

Solution

The row passes this regression check, but the conclusion is bounded to the tested prefix, model, dtype, environment, and tolerance.

Q8. Calculate float32 cache bytes

For $B=2,L=4,h=8,T=16,d_h=32$ , calculate K/V scalar count and float32 payload bytes.

Your answer

Answer it first, then check.

Hint

Calculate

2LBhTd_h

Solution

2(4)(2)(8)(16)(32)=65,536

scalars and 262,144 bytes.

Q9. Choose batched stopping behavior

Row A reaches <eos>. Row B has not stopped but already has cache length $C$ . What stopping reasons should be reported?

Your answer

Answer it first, then check.

Hint

Do not call a context limit an EOS prediction.

Solution

Report eos for A and context for B.

Q10. Diagnose a cache trace

The cache length grows from 2 to 4 after one consumed token, layer 2's cached values exactly equal layer 1's, and a different request changes the logits. Name the three failures.

Choose the three failures

Duplicate append, wrong-layer cache reuse, and cross-request cache contamination.

Greedy decoding, low temperature, and top-k filtering.

A head-count mismatch, a value-width mismatch, and context overflow.

Select one choice, then check.

Hint

Audit how often, where, and for whom cache state is written.

Solution

The token was appended twice, layer 2 reused layer 1 state, and cache storage is incorrectly shared across requests.

Q1. Trace a greedy generation step

Q2. Apply temperature

Q3. Distinguish prefill and decode

Q4. Calculate cache shape and count

Q5. Calculate one attention score

Q6. Continue the position index

Q7. Interpret logit error

Q8. Calculate float32 cache bytes

Q9. Choose batched stopping behavior

Q10. Diagnose a cache trace

Pause and reflect