RoPE Rotates Query and Key Pairs

RoPE splits projected queries and keys into coordinate pairs and rotates each pair using its position and frequency. Calculate the geometry, verify norm preservation, and record pairing, sign, origin, scaling, and precision conventions.

Rotary Position Embedding, or RoPE, inserts position information after the query and key projections. It divides query and key features into 2D pairs and rotates each pair by an angle determined by the token position and that pair's frequency.

The standard formulation taught here rotates queries and keys, not values. A codebase may arrange paired coordinates differently, so we will state the layout before calculating anything.

Start with One 2D Rotation

For column vectors, define

R(\phi)= \begin{bmatrix} \cos\phi&-\sin\phi\\ \sin\phi&\cos\phi \end{bmatrix}.

At position $m$ , one query pair $q$ becomes

q_m=R(m\theta)q,

where $\theta$ is the frequency assigned to that pair.

Let $q=[1,0]^T$ , $\theta=\pi/4$ , and $m=2$ . The angle is $\pi/2$ , so

q_2=R(\pi/2)[1,0]^T=[0,1]^T.

The rotation changes direction while preserving length.

Extend the Operation Across the Head Width

For an even query-key head width $d_k$ , this chapter pairs adjacent coordinates:

(0,1),(2,3),\ldots,(d_k-2,d_k-1).

Each pair has its own frequency. A common RoPE schedule follows the same broad fast-to-slow pattern as sinusoidal encodings. The complete query at position $m$ receives block-diagonal rotations: one 2D block for every coordinate pair.

For a four-coordinate query,

q=[q_0,q_1,q_2,q_3],

the first pair may turn quickly while the second turns slowly. The operation does not mix the two pairs with each other.

Verify the Geometry

Rotate adjacent query-key pairs

The code applies one stated adjacent-pair convention and checks that every pair preserves its Euclidean length.

The first pair becomes approximately $[0,1]$ . The second pair rotates through the smaller angle $\pi/8$ .

Conventions That Must Agree

Two implementations can both be called RoPE and still disagree because of:

adjacent-coordinate versus split-half pairing;
the sign used by the 90-degree partner operation;
whether positions begin at 0 or another offset;
the frequency base and any context-length scaling;
which fraction of the query-key width is rotated;
the dtype used to calculate angles at large indices.

Shape checks will not detect these differences. A reliable port compares selected rotated values and attention logits under the same convention.

Q1. Rotate one query pair

Using the rotation matrix on this page, rotate $q=[1,0]^T$ by angle $\phi=\pi/4$ . What is the second coordinate?

Your answer

Compute it first, then check your number.

Hint

The rotated vector is

[\cos\phi,\sin\phi]^T

Solution

The second coordinate is

\sin(\pi/4)=\sqrt{2}/2\approx0.7071

Reference

Jianlin Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021.