Caching Trades Recalculation for Growing Memory

Without a cache, each generation step runs the complete growing prefix through every decoder block again. With a cache, each new token reuses earlier keys and values and runs only its own residual path through the blocks.

Calculate Cache Memory

For standard multi-head attention:

stored scalars. The factor 2 represents keys and values.

For the frozen model at $B=1$, $L=2$, $h=2$, and $d_h=2$:

At $T=4$, this is 64 scalars or $64(4)=256$ bytes in float32.

These numbers exclude tensor metadata, allocator overhead, model weights, temporary scores, activations for the current token, and framework state.

State What Work Remains

At decode length $T$, each new query still compares with $T$ keys and mixes $T$ values. Dense cached attention therefore grows linearly per new token. Across a generated sequence, that attention work remains quadratic in total length.

Caching avoids recomputing previous tokens' projections, MLPs, normalizations, and residual updates at every generation step. Under a simple dense-operation count, repeatedly running the full growing sequence incurs cubic total attention work, while cached one-row decode reduces that repeated part to quadratic. Actual latency also depends on kernels, memory bandwidth, batch, hardware, and framework overhead.

Research Boundary

• Pope et al., Efficiently Scaling Transformer Inference

That work analyzes and measures specified large models on TPU v4 systems. It supports careful inference trade-off analysis; its latency and utilization numbers do not describe this tiny CPU Python implementation.