Audit Generation and Cache State

Debug generation in dependency order: checkpoint and logits, cache tensors, positions and ownership, numerical equivalence, policy state, then stopping. Use invariants and failure signatures instead of judging output alone.

Generation failures can come from the model, policy, cache, positions, stopping logic, or request bookkeeping. Separate those layers before tuning output.

1. Establish the Uncached Reference

load the intended checkpoint and vocabulary;
set evaluation and inference mode;
run the complete prompt without a cache;
record its final logits row.

If this result is already wrong, a cache cannot repair it.

2. Audit Cache Construction

verify one K and V tensor per layer;
verify $(B,h,t,d_h)$ axes and sequence-axis append;
verify layer ownership;
verify cache length equals consumed input count;
compare each new K/V record with full-prefix projections.

3. Audit Position and Request State

continue positions from prompt length;
keep cache and counter per request;
reject stale caches after parameter changes;
never combine one prompt's cache with another prompt's token stream;
reject context overflow before reading a nonexistent position row.

4. Audit Policy after Logit Equivalence

Hold logits fixed. Check temperature domain, top- $k$ range, renormalization, tie behavior, and random-generator state. A different sampled token does not by itself prove a cache error.

Failure Matrix

Observation	First check
mismatch begins at token plus position	position index or token ownership
layer 1 matches, layer 2 differs	layer 2 cache source and append
cache length grows by two	duplicate append or duplicated prompt token
batch row changes when another request arrives	cache mixing across batch or request axes
greedy differs but logits match	policy or tie rule
sampled runs differ but probabilities match	seed and RNG call order
cache works until weights update	stale projected K/V
generated token cannot be fed back	context limit reached

The executed wrong-position and wrong-layer failures preserve valid tensor ranks while producing logit errors 23.0252 and 3.6990 respectively.

Q1. Diagnose two cache failures

Full and cached logits match through layer 1. At layer 2, cached length is correct but values equal layer 1's cache. A second request also changes the first request's cached logits. Identify both failures.

Choose both failures

Wrong-layer cache reuse and cross-request cache contamination.

Temperature scaling and a top-k renormalization error.

Context overflow and an EOS stopping error.

Select one choice, then check.

Hint

Ask who owns each cache along two separate axes.

Solution

Layer 2 is reading layer 1's K/V records, and request-local state is being shared or overwritten across requests.

Inspection Comes Next

Once cached and uncached generation agree, Chapter 9 can intervene on attention, residual updates, activations, and logits without confusing an implementation mismatch with a model behavior.