Why Attention Became Attractive

Attention offered a different path.

Instead of forcing all past information through one recurrent hidden state, an attention mechanism can compare the current position with earlier positions more directly.

This does not make attention free. It has its own cost. But it changes the information route:

RNN: past -> hidden -> hidden -> hidden -> current
attention: current position can look back at selected earlier positions

That direct comparison is one reason attention became central to modern language models.

Exercise

Does attention reduce the need to carry all information through one fixed hidden state? Answer 1 for yes or 0 for no.