State-Space Models Are Alternatives, Not Transformer Variants
Contrast explicit query-key interactions and K/V history with recurrent state updates, while preserving implementation and empirical boundaries.
A state-space sequence model and a Transformer can solve similar sequence tasks while using different mechanisms. Calling both “attention variants” hides the important change.
Compare the State Contracts
A simplified recurrent state update has the form
The current state summarizes earlier inputs. In a selective state-space model, some update parameters depend on the current input. A causal Transformer instead retains position records or their K/V projections and computes explicit query-key interactions.
| Property | Causal self-attention | Recurrent state-space mixer |
|---|---|---|
| cross-position mechanism | explicit query-key/value mixing | state update and readout |
| main retained decode state | per-layer K/V history | recurrent state |
| attention matrix | conceptually present in full pass | absent |
| shared goal | contextual sequence records | contextual sequence records |
The shared goal does not make the internal contracts equivalent.
Read Performance Claims with Their Implementation
Mamba combines a selective state-space mechanism with a hardware-aware parallel algorithm. Its paper reports results for specified scales, modalities, implementations, and benchmarks. Those results do not show that attention is obsolete, nor do they reduce the method to the two equations above.
Q1. Classify the sequence mixer
A model replaces pairwise query-key interactions with an input-dependent recurrent state update. Is this best described as a sparse-attention pattern or an alternative sequence mixer?
Select one choice, then check.
Hint
Solution
Preserve the Curriculum Boundary
This comparison supplies vocabulary and a contract. Deriving state-space models, discretization, selective scans, kernels, and full empirical evaluation belongs to a later frontier-architectures subject.