Review

Review the generation contract, model-policy-cache separation, prefill and decode shapes, position state, cache memory, numerical equivalence evidence, token traces, and dependency-ordered audit.

Generation repeats one contract:

\text{prefix}\rightarrow\text{final logits} \rightarrow\text{policy}\rightarrow\text{next token} \rightarrow\text{extended prefix}.

Stop when the policy selects <eos>, when another input would exceed context, or when another declared rule applies. Record the stopping reason.

Separate Model, Policy, and Cache

The model and checkpoint determine logits.
The decoding policy determines how a token is selected from logits.
The KV cache changes which stable computations are reused.

A correct cache preserves logits within tolerance. A different policy may select a different token from the same logits.

Reconstruct Prefill and Decode

Prefill processes $P$ prompt positions and creates $P$ key and value records per layer and head. Decode consumes one new input token, creates one new query, key, and value, appends K/V, and calculates one new logits row.

At cached length $t$ :

q_t,k_t,v_t:(B,h,1,d_h), \qquad K,V:(B,h,t+1,d_h).

The current query attends across the full valid cache. Previous queries are not needed for the next standard attention output.

Reconstruct Position State

The next zero-based position equals the number of consumed input tokens. A prompt of length $P$ uses positions 0 through $P-1$ ; its first fed-back token uses position $P$ .

The frozen model accepts four input positions. Its fourth logits row can predict a fifth returned token, but the fifth token cannot become another input.

Reconstruct Cache Memory

Standard multi-head K/V payload contains

2LBhTd_h

scalars. For the frozen model, this is $16T$ scalars or $64T$ float32 bytes. At $T=4$ , the payload is 256 bytes, excluding metadata, weights, and temporary values.

Caching saves repeated prefix projections and block computation. The new query still attends across all cached positions, so dense attention work per new token grows with context length.

Reconstruct the Equivalence Evidence

All 16 prefixes of the four controlled sequences pass:

maximum absolute logit error: $4.7684\times10^{-6}$ ;
acceptance tolerance: $10^{-5}$ ;
wrong-position error: 23.0252;
wrong-layer-cache error: 3.6990.

Greedy generation returns <bos> A B D <eos>. Seeded top-3 sampling returns <bos> Y X Y <eos>. These token traces test deterministic execution and random state; they do not measure language quality.

Audit in Dependency Order

checkpoint, vocabulary, model mode, and uncached logits;
per-layer K/V values, axes, append, and length;
position index and request ownership;
cached-versus-full intermediate and final values;
policy settings and random state;
EOS, context, and batched stopping behavior.

Chapter 9 assumes this computation is correct and asks what controlled observations and interventions can reveal about it.