Direct Access to Earlier Positions

Attention lets a position compare with earlier positions more directly.

Instead of relying only on a compressed running state, the model can assign weights to previous token representations and combine them.

This does not mean attention understands everything. It means the architecture provides a better route for using earlier information.

Exercise

If a token attends to 4 earlier positions, how many earlier positions are being read from?