Relative Position Changes Pair Interactions

Absolute methods attach information to one index $i$. A relative method can instead change the interaction between query position $i$ and key position $j$ according to an offset such as $j-i$.

Offsets preserve both distance and direction:

• $j-i=-2$ means the key is two positions before the query;
• $j-i=0$ means query and key share an index;
• $j-i=3$ means the key is three positions after the query.

Changing the sign convention is valid only if every table lookup, equation, test, and cached index uses the same convention.

One Concrete Relative-Attention Design

Shaw, Uszkoreit, and Vaswani proposed adding a learned relation vector to each query-key interaction. In one-head notation, a score can be written

Their construction can also change the value reading:

Here $a^K_r$ and $a^V_r$ are learned vectors for relative offset $r$. This is one relative-position mechanism, not a formula shared by every later method.

Equal Content Can Receive Different Scores

Let one query and three content keys all be

with $d_k=2$. For query position $i=2$, the offsets $j-i$ are $-2,-1,0$. Let

The three scores are

Content alone could not distinguish the equal keys. The relative terms make their direction and distance affect matching.

Clip or Bucket the Offset Range Explicitly

A learned relation table is often finite. One policy clips offsets to a range $[-K,K]$:

With $K=4$, offsets $-9$ and $-4$ use the same table row. This bounds parameter count but discards exact distance beyond the boundary. Other methods use buckets whose widths grow with distance. The policy is part of the model, not an implementation detail to omit.

Relative Does Not Mean Translation-Invariant Model

If every query and key index shifts by the same amount, $j-i$ remains the same. That algebraic property concerns this position term. Padding boundaries, causal visibility, sequence starts, special tokens, data patterns, and later layers can still make complete model behavior depend on absolute location.

Reference

• Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani, Self-Attention with Relative Position Representations, NAACL 2018.