Sinusoidal Position Encodings

Fixed sine-cosine pairs change at several frequencies and add a repeatable vector at each index. Calculate four coordinates, interpret a fixed offset as pairwise rotations, and separate formula range from trained length generalization.

The original Transformer used a fixed position vector built from sine and cosine waves. The vector contains no learned entries. Different coordinate pairs change at different rates, giving each position a repeatable pattern across several frequencies.

For even coordinate $2k$ and the following odd coordinate $2k+1$ ,

PE(pos,2k)=\sin\left(\frac{pos}{10000^{2k/d_{model}}}\right),

PE(pos,2k+1)=\cos\left(\frac{pos}{10000^{2k/d_{model}}}\right).

Position is usually counted from 0, but an implementation must state its index origin rather than assume it.

Calculate Four Coordinates

Let $d_{model}=4$ . For $k=0$ , the denominator is 1. For $k=1$ , it is $10000^{1/2}=100$ . The position vector is therefore

PE(pos)=[\sin(pos),\cos(pos),\sin(pos/100),\cos(pos/100)].

Rounded to six decimal places:

Position	Coordinate 0	Coordinate 1	Coordinate 2	Coordinate 3
0	0	1	0	1
1	0.841471	0.540302	0.010000	0.999950
2	0.909297	-0.416147	0.019999	0.999800

The first pair turns quickly. The second pair moves slowly over these three positions. At a realistic model width, many pairs span many rates.

Inspect sinusoidal frequencies

Change the positions, model width, or base and inspect each sine-cosine pair.

A Fixed Offset Is a Rotation Within Each Pair

For one frequency, write the pair as

u(pos)=[\sin(\omega pos),\cos(\omega pos)].

The angle-addition formulas express $u(pos+\Delta)$ as a fixed linear transformation of $u(pos)$ whose coefficients depend on $\omega\Delta$ . Each frequency pair therefore responds to the same positional offset through its own 2D rotation.

This property helps make relative offsets accessible to learned projections. It does not mean a trained network must infer every distance perfectly.

Add the Vector Once at the Input Boundary

In the original input construction,

x_i=E[t_i]+PE(i).

The position vector has width $d_{model}$ and is added before the stack of Transformer blocks. Later layers can transform and combine the resulting features. Re-adding the same vector at every layer would define a different architecture.

Computable Is Not the Same as Generalized

The formula can calculate $PE(pos)$ for an index larger than any index used in training. A learned table cannot index an absent row. This is a real implementation difference, but it does not by itself prove better behavior on longer sequences. The learned model may still rely on training-length patterns, attention distributions, or data statistics that fail outside their observed range.

Q1. Calculate a sinusoidal coordinate

For $d_{model}=4$ , what is coordinate 2 of $PE(2)$ ? Give the answer to at least four decimal places.

Your answer

Compute it first, then check your number.

Hint

For coordinate 2,

k=1

and the denominator is

10000^{2/4}=100

Solution

PE(2,2)=\sin(2/100)=\sin(0.02)\approx0.019999

Reference

Ashish Vaswani et al., Attention Is All You Need, 2017.