Learned Absolute Position Vectors

A learned table adds one trainable model-width vector to each supported index. Calculate repeated-token examples, count parameters, and audit maximum length, padding, index origins, shifts, swaps, and zero-position ablations.

A learned absolute-position method stores one trainable vector for each supported position. Before the first Transformer block, the model combines the token vector and the position vector for that row.

If token ID $t_i$ has embedding $E[t_i]$ and position $i$ has vector $P[i]$ , then the initial representation is commonly

x_i=E[t_i]+P[i].

Both vectors must have width $d_{model}$ so addition preserves the layer interface.

The Same Token Can Enter Differently

Suppose the token blue has embedding

e=[1,0,1,0]

and appears at positions 0 and 2. Let

p_0=[0.1,0,0,0.2],\qquad p_2=[-0.1,0.3,0,0].

The two input records are

x_0=e+p_0=[1.1,0,1,0.2],

x_2=e+p_2=[0.9,0.3,1,0].

The token lookup is identical, but the records supplied to the projections are not. Training can adjust $p_0$ and $p_2$ through the same loss that adjusts the token embeddings and the rest of the model.

Parameter Shape and Index Range

For a table supporting positions $0$ through $L_{max}-1$ ,

P:L_{max}\times d_{model}.

It contributes $L_{max}d_{model}$ trainable entries. A batch of position IDs with shape $(B,T)$ indexes the table and returns position vectors with shape $(B,T,d_{model})$ .

The table has no row at index $L_{max}$ . Extending the context therefore requires a declared policy, such as enlarging and training the table or using a different method. Silently wrapping indices would make distant positions share rows and would change the model's position convention.

Position IDs Must Match the Records

For an unpadded three-token sequence, position IDs may be

[0,1,2].

Padding makes the convention more visible. Consider two batch rows:

[PAD, PAD, A, B]
[C,   D,   E, F]

Possible policies include assigning the real tokens in the first row IDs $[0,1]$ or retaining their physical columns $[2,3]$ . Either can be implemented, but training, evaluation, masks, and cached decoding must agree. Padding tokens must also be excluded by the padding mask; a convenient position ID does not replace visibility masking.

Controlled Changes Reveal the Role

Useful ablations include:

set every $P[i]$ to zero while keeping token embeddings and all other parameters fixed;
swap two position rows while leaving the token sequence unchanged;
use the same position row at every index;
shift every valid position ID by one, if the table range permits it.

These interventions answer different questions. Zeroing removes the explicit absolute vectors. Swapping corrupts selected index associations. A global shift tests sensitivity to the index origin. None isolates position information that may also arise from a causal mask or the data.

Q1. Count learned position entries

A learned absolute-position table supports $L_{max}=512$ positions with $d_{model}=768$ . How many trainable scalar entries does the table contain?

Your answer

Compute it first, then check your number.

Hint

Count the entries in a matrix with one row per supported position.

Solution

512(768)=393{,}216

trainable entries.

Learned Does Not Mean Arbitrary

Learned rows do not start with a guaranteed notion of distance or direction. Their useful structure must arise through training. Conversely, a fixed formula that can calculate a vector for a new index does not guarantee that a model trained only on shorter sequences will use that vector successfully. Parameter availability and length generalization are separate questions.

Reference

Ashish Vaswani et al., Attention Is All You Need, 2017. The original Transformer used sinusoidal encodings and reported similar results for a learned positional alternative in its stated translation experiments.