LLM Primer

Introduction

Attention becomes natural after recurrence.

RNNs carry information forward through a hidden state. LSTMs and GRUs improve that path with gates. Attention changes the route: a position can read from other positions using learned weights.

attention reads earlier positions with learned weightstheweight 0.10bookweight 0.55wasweight 0.25oldweight 0.10weighted reading replaces one fixed hidden summary with direct comparison
Attention lets a position combine information from other positions instead of relying only on a recurrent state.

What this chapter covers

  • sequential bottleneck;
  • fixed hidden-state bottleneck;
  • direct access to earlier positions;
  • alignment intuition;
  • attention as weighted reading.