Summary and Revision Notes

Core ideas

• Long dependencies appear in agreement, reference, code, and extended context.
• Vanishing gradients make distant training signals weak.
• Exploding gradients make updates unstable.
• A fixed hidden state can become a memory bottleneck.
• Attention became attractive because it gives more direct access to earlier positions.

Check yourself

• Can you name one language example with a long dependency?
• Can you explain why repeated multiplication by small values weakens gradients?
• Can you explain why a fixed hidden state may overwrite useful information?