Chapter 15
From Recurrence to Attention
Sequential bottlenecks, fixed hidden-state bottlenecks, direct access to earlier positions, alignment intuition, weighted reading, and long-context motivation.
What this chapter does
Attention becomes natural once the reader has seen the limits of fixed windows and recurrent hidden states.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Why attention follows naturally after recurrence and gated recurrence.
- 02Sequential Bottleneck
Why recurrent order limits parallelism.
- 03Fixed Hidden-State Bottleneck
Why one fixed state can compress too much history.
- 04Direct Access to Earlier Positions
Attention as a more direct route to previous token representations.
- 05Alignment Intuition
Attention weights as soft alignment over useful positions.
- 06Attention as Weighted Reading
Combining representations with learned weights.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What attention motivation establishes before Transformers.
Summary and Revision NotesA compact review of recurrence bottlenecks, direct access, alignment, and weighted reading.
ExercisesChapter-level practice for attention motivation and weighted reading.
HintsLow-spoiler nudges for the Chapter 15 exercises.
SolutionsExplained solutions for the Chapter 15 exercises.
Before moving on
- Explain attention as a response to earlier limitations.
- Compute a tiny weighted average over previous token representations.
- Understand why attention changes parallelism and long-context behavior.
Where this leads
- Transformers
- Language Modeling as a Foundation for LLMs