LLM Primer

Introduction

Recurrent neural networks process a sequence one step at a time.

At each step, the model reads the current token and updates a hidden state. The hidden state is the model's running summary of what it has seen so far.

This was the main neural sequence idea before attention became dominant. Understanding recurrence makes attention easier to motivate: it shows both the power and the bottleneck of carrying information through a single state.

an RNN reuses one update rule across sequence positionsRNNtoken 1RNNtoken 2RNNtoken 3hidden statehidden stateunrolling shows repeated computation, not separate models
The same recurrent cell is applied at each step while the hidden state carries information forward.

What this chapter covers

  • sequence processing step by step;
  • hidden state;
  • shared parameters across time;
  • unrolling through time;
  • backpropagation through time.