Self-attention compares token representations and combines values. The same projection matrices are used at every position. If an unmasked input is reordered, content-only attention follows that reordering: its output rows are reordered in the same way. The operation has no separate record of which row was first, second, or third.
Decoder attention already contains one position-dependent object: the causal mask. A query at index 0 may read one position, while a query at index 5 may read six. The mask therefore breaks the full reordering symmetry and can supply some implicit position information. It does not provide the same mechanism as an explicit position method.
This distinction matters. It is too broad to say that every Transformer without position embeddings treats its input as an unordered bag. It is also too broad to infer that explicit position methods are unnecessary because some causal models work without them.
This chapter compares four ways to make order or distance affect a Transformer:
- learned absolute vectors added to token representations;
- fixed sinusoidal vectors added at each position;
- relative information inserted into pairwise attention interactions;
- RoPE rotations or ALiBi biases applied to queries, keys, or scores.
The location of the change is part of the method. Adding a vector to the input, rotating query-key coordinate pairs, and biasing a score before softmax are not interchangeable operations.