Sequential Bottleneck

Recurrent models process tokens in order. Step t + 1 depends on step t.

That order is natural for sequences, but it limits parallelism. Later positions must wait for earlier recurrent updates.

Attention-based models changed this by allowing many position-to-position comparisons to be computed in parallel.

Exercise

If an RNN must process 6 tokens step by step, how many ordered steps are needed?