Gated Recurrence as Attention Precursor

Gated recurrence is not attention, but it helps explain why attention was needed.

LSTMs and GRUs improved the memory path inside recurrent models. They still processed sequences step by step. Information from a far earlier position still had to travel through many updates.

Attention changed the route. Instead of only improving the memory carried forward, it allowed positions to compare with other positions more directly.

That is why this chapter belongs before attention: gates show a serious attempt to manage memory, and their limits motivate a different mechanism.

Exercise

Do LSTMs and GRUs still process sequences step by step? Answer 1 for yes or 0 for no.