Review

Review architecture cards, sparse visibility, long-context evidence, KV-head sharing, MoE ledgers, multimodal and state-space contracts, scaling, compute allocation, and system boundaries.

This chapter supplied a method for reading unfamiliar model claims. Begin with the changed contract, preserve the invariant core, calculate what follows from the definition, and keep measured results inside their evidence boundary.

Reconstruct the Architecture Card

Record five fields:

changed component;
invariant core;
analytical consequence;
empirical source boundary;
later subject that owns the missing depth.

Input representation, sequence mixing, per-position transformation, training, and the surrounding system are separate contracts.

Reconstruct the Calculations

Dense causal attention has

E_{dense}=\frac{T(T+1)}{2},

while a causal local window with at most $w$ preceding tokens has

E_{local}=\sum_{i=0}^{T-1}\min(i+1,w+1).

For $T=8,w=2$ , the counts are 36 and 21. Fewer edges are guaranteed; preserved task quality is not.

A K/V cache with $h_{kv}$ heads contains

2LBh_{kv}Td_h

scalars. Under the chapter values, MHA, two-head GQA, and MQA contain 8192, 2048, and 1024 scalars.

In the top-1 MoE ledger, 320 parameters are stored while one selected expert contains 76 parameters. This does not erase routing, communication, capacity, or imbalance costs.

Reconstruct the Evidence Limits

accepting a long prompt is not the same as retrieving distant evidence;
a patch-token Transformer is not every multimodal architecture;
a state-space mixer is an alternative contract, not sparse attention;
two log-space points determine a fit but do not validate extrapolation;
equal simplified compute does not imply equal quality;
a product improvement does not identify a base-model improvement.

Reconstruct the Complete System

Keep base architecture, pretraining, post-training, inference policy, retrieval/tools, evaluation, product policy, and infrastructure visible. Name every changed layer before attributing a result.

The core path is now complete: language-model foundations led to attention, Transformer blocks, a decoder-only model, training, generation, caching, interpretation, and finally a disciplined map of variants and system boundaries. The next step is to build and report the complete tiny-decoder project without relaxing any of these contracts.