Introduction

RNNs can, in principle, carry information across a sequence. In practice, long dependencies are hard.

Information must pass through many hidden-state updates. Gradients must also travel through those repeated steps during training. Along the way, signals can shrink, grow, or be overwritten.

This chapter explains the problem that gated RNNs tried to solve and attention later attacked more directly.

What this chapter covers

• why long dependencies matter;
• vanishing gradients;
• exploding gradients;
• memory bottlenecks;
• why attention became attractive.