Chapter 10

Variants, Scaling, and Boundaries

Map important Transformer variants and system claims without turning names into explanations. Separate architecture, data, compute, pretraining, post-training, inference, retrieval, tools, and evaluation, then identify the later subject that owns each missing layer.

A new model name does not always mean a new foundation. A system may change which tokens can interact, share key and value heads, route tokens through different MLPs, accept another modality, or replace attention altogether. To understand the change, identify the contract that changed and the parts that remained fixed.

This chapter closes the subject with a method for reading architecture and scaling claims. It uses small calculations rather than a catalogue of model names. You will compare:

  • dense and sparse visibility graphs;
  • advertised context length and demonstrated long-range use;
  • multi-head, grouped-query, and multi-query K/V caches;
  • total and active parameters in a mixture-of-experts layer;
  • image tokenization and image-text fusion;
  • attention-based and state-space sequence mixing;
  • measured scaling trends and unsupported extrapolation;
  • model size, training tokens, and a fixed compute budget.

The final lesson separates a base model from post-training, retrieval, tools, evaluation, product policy, and infrastructure. These layers work together in an LLM product, but they are not one mechanism and do not belong in one architecture claim.

The calculations are reproduced by the chapter script. Its scaling values are deliberately illustrative. Research results retain the models, data, hardware, tasks, and measured ranges reported by their sources.

After this chapter

  • Identify the core computation retained by a Transformer variant.
  • Separate architecture, data, compute, training, inference, and evaluation claims.
  • Choose the correct later subject for generative, RL, frontier, or engineering depth.

Lessons

  1. 01
    Describe a Variant by Its Changed Contract

    Compare model variants by input, sequence mixer, feature transformation, training, and surrounding-system contracts.

    1 exercise
  2. 02
    Sparse Attention Changes the Visibility Graph

    Compare dense and sparse causal attention graphs with exact edge counts, dependency paths, and research boundaries.

    1 exercise
  3. 03
    Long Context Is More than a Larger Length Number

    Audit long-context model claims through six distinct evidence layers and a complete context contract.

    1 exercise
  4. 04
    Key-Value Head Sharing Changes Inference Memory

    Compare MHA, GQA, and MQA head-sharing contracts, KV-cache tensor shapes, scalar counts, and evidence limits.

    1 exercise
  5. 05
    Mixture of Experts Separates Total and Active Parameters

    Understand sparse mixture-of-experts routing with an exact parameter ledger and explicit system-cost boundaries.

    1 exercise
  6. 06
    Multimodal Transformers Need an Input and Fusion Contract

    Compare image patch tokenization and multimodal fusion contracts with exact shapes and bounded research examples.

    1 exercise
  7. 07
    State-Space Models Are Alternatives, Not Transformer Variants

    Compare Transformer attention and selective state-space sequence mixers by mechanism, retained state, and evidence scope.

    1 exercise
  8. 08
    Scaling Laws Are Empirical Fits within a Measured Regime

    Calculate and audit empirical neural language-model scaling fits without turning them into universal laws.

    1 exercise
  9. 09
    Compute-Optimal Training Couples Model, Data, and Budget

    Analyze compute-optimal training through model size, token count, fixed budgets, data quality, and bounded scaling evidence.

    1 exercise
  10. 10
    Map the Complete LLM System and the Next Subjects

    Map the complete LLM system and distinguish model architecture from training, retrieval, tools, evaluation, policy, and serving.

    1 exercise

Review and practice

  1. •
    Review

    Review Transformer variants, scaling evidence, compute allocation, architecture contracts, and complete LLM system boundaries.

  2. •
    Exercises

    Solve integrated exercises on Transformer variants, scaling, memory, parameters, multimodality, alternatives, compute, and LLM systems.

Chapter progress