RoPE Dot Products Depend on Relative Offset

Absolute query and key rotations combine into a dot product determined by their relative offset. Derive the identity, test simultaneous shifts, and preserve position indices across cached and full-prefix execution.

RoPE uses absolute indices to choose two rotations, but their query-key dot product can be written using the difference between those indices. This is the mechanism's central relative-position property.

Derive the Identity

Rotate query $q$ at position $m$ and key $k$ at position $n$ :

q_m=R(m\theta)q,\qquad k_n=R(n\theta)k.

Because a rotation matrix is orthogonal and $R(\alpha)^TR(\beta)=R(\beta-\alpha)$ ,

q_m^Tk_n =q^TR(m\theta)^TR(n\theta)k =q^TR((n-m)\theta)k.

The score contribution from this coordinate pair depends on relative offset $n-m$ , using the key-minus-query convention.

Calculate a Small Case

Let $q=k=[1,0]^T$ and $\theta=\pi/4$ .

Query position $m$	Key position $n$	Offset $n-m$	Dot product
0	0	0	1
0	1	1	$\sqrt{2}/2$
1	3	2	0
4	5	1	$\sqrt{2}/2$

The pairs $(0,1)$ and $(4,5)$ have the same offset and therefore the same RoPE contribution for the same unrotated $q$ and $k$ .

Shift Both Indices Together

If both positions increase by $c$ ,

(n+c)-(m+c)=n-m.

The pairwise RoPE dot product is unchanged. This does not prove that the whole model is invariant to shifting a sequence. Causal and padding boundaries, special tokens, token content, finite context, numerical precision, and later operations remain part of the computation.

Test relative RoPE logits

The program compares position pairs with equal offsets and exposes an inconsistent key-position bug.

Cached Decoding Must Continue the Position Count

During generation, previously computed keys retain the rotations associated with their original indices. A new query and key must use the next absolute index under the same convention. Restarting the new token at position 0 can leave every tensor shape valid while changing its logits.

Chapter 8 will build the key-value cache. For now, retain this invariant:

Cached and uncached execution must assign the same position index to every query and key representing the same token occurrence.

A Relative Score Property Is Not a Universal Length Guarantee

RoPE can calculate rotations at indices beyond those seen during training. Performance at those indices depends on learned behavior, frequency schedules, precision, scaling methods, task, data, and evaluation. The algebraic identity does not settle those empirical questions.

Q1. Use the relative RoPE identity

For one RoPE pair, $q=k=[1,0]^T$ and $\theta=\pi/4$ . The query position is $m=7$ and key position is $n=9$ . What is their rotated dot product?

Your answer

Compute it first, then check your number.

Hint

Use

\cos((n-m)\theta)

for these two unrotated vectors.

Solution

n-m=2

, giving

\cos(2\pi/4)=\cos(\pi/2)=0

Reference

Jianlin Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021.