Chapter 12
Long-Range Dependency Problems
Long dependencies, vanishing gradients, exploding gradients, memory bottlenecks, and why attention became attractive.
What this chapter does
Processing tokens in order does not guarantee that distant information remains usable. This chapter makes the limitation visible.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Why long dependencies expose recurrent-model limits.
- 02Why Long Dependencies Matter
Agreement, reference, code, and extended context often depend on distant tokens.
- 03Vanishing Gradients
Repeated paths can make distant gradient signals very small.
- 04Exploding Gradients
Repeated paths can make gradients unstable and too large.
- 05Memory Bottlenecks
A fixed hidden state must compress useful past information.
- 06Why Attention Became Attractive
Attention gives a more direct route to earlier positions.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What long-range dependency problems establish before gated recurrence.
Summary and Revision NotesA compact review of long dependencies, vanishing gradients, exploding gradients, memory bottlenecks, and attention motivation.
ExercisesChapter-level practice for long-range dependency and gradient-path reasoning.
HintsLow-spoiler nudges for the Chapter 12 exercises.
SolutionsExplained solutions for the Chapter 12 exercises.
Before moving on
- Explain why ordered processing can still lose distant information.
- Recognize gradient problems through time.
- Use the limitation to motivate gated recurrence.
Where this leads
- LSTMs and GRUs
- From Recurrence to Attention