Summary and Revision Notes

Core ideas

• Recurrence has a sequential bottleneck.
• A fixed hidden state can bottleneck memory.
• Attention gives more direct access to earlier positions.
• Attention weights can be read as soft alignment.
• Attention performs weighted reading over representations.

Check yourself

• Can you explain why recurrence limits parallelism?
• Can you compute a tiny weighted-reading sum?
• Can you explain why attention belongs before Transformers?