Attention as Weighted Reading

Attention forms a weighted combination of representations.

If the weights are:

0.5, 0.3, 0.2

then the first representation contributes most, the second less, and the third least. The weights usually sum to 1 for one attention distribution.

This weighted reading prepares the Transformer idea: many positions can read from many other positions through learned attention patterns.

Exercise

What is the sum of attention weights 0.5, 0.3, and 0.2?