Review

Review reordering and causal-mask boundaries, absolute and relative position equations, RoPE geometry, ALiBi score bias, mechanism locations, context claims, and a ten-step audit order.

Begin with the Attention Setting

For unmasked content-only self-attention and a row permutation $P$ ,

\operatorname{Attention}(PX)=P\operatorname{Attention}(X).

This is permutation equivariance, not invariance. The output records follow the reordered input records.

A fixed causal mask is tied to index order. In general,

P(S+M)P^T\ne PSP^T+M.

Causal visibility can therefore provide implicit position information. This qualifies claims about Transformers without explicit position methods; it does not make all explicit methods unnecessary.

Absolute Position Methods

A learned absolute table uses

x_i=E[t_i]+P[i],\qquad P:L_{max}\times d_{model}.

It adds $L_{max}d_{model}$ trainable entries and requires a policy beyond the available table rows.

The original sinusoidal method uses fixed pairs:

PE(pos,2k)=\sin\left(\frac{pos}{10000^{2k/d_{model}}}\right),

PE(pos,2k+1)=\cos\left(\frac{pos}{10000^{2k/d_{model}}}\right).

Each pair uses a different frequency. A fixed offset acts as a frequency-specific 2D rotation. The formula can compute a new index, but that does not guarantee that a trained model generalizes to the corresponding sequence length.

Relative Interaction Methods

One Shaw-style relative score is

e_{ij}=\frac{q_i^T(k_j+a^K_{j-i})}{\sqrt{d_k}}.

The signed offset convention and any clipping or bucketing rule are part of the model definition.

RoPE rotates query and key coordinate pairs. Under this chapter's column-vector convention,

(R(m\theta)q)^T(R(n\theta)k) =q^TR((n-m)\theta)k.

Absolute indices select rotations, while the pairwise dot product has a relative-offset form. Pair layout, sign, index origin, frequencies, scaling, rotated fraction, and numerical precision must agree across implementations.

ALiBi adds a finite distance penalty to allowed scores:

e^{(r)}_{ij} =\frac{q_i^{(r)\top}k_j^{(r)}}{\sqrt{d_k}}-m_r(i-j).

It is a recency bias, not a causal mask and not a hard attention window.

Compare the Operation Location

Method	Changed object
Learned absolute	input representation
Sinusoidal	input representation
Shaw-style relative	pairwise score and optionally value
RoPE	projected query and key pairs
ALiBi	attention score before softmax

Methods that produce the same final tensor shape can still implement different functions.

Audit Order

Record token IDs, padding, special tokens, and position IDs.
State index origin and signed-offset convention.
Keep causal and padding masks separate from finite score biases.
Verify absolute additions, if used.
Verify selected Q/K rotations or relation lookups, if used.
Record content scores and position contributions separately.
Check combined scores, softmax axis, row sums, and forbidden zeros.
Compare changed, zeroed, shifted, or swapped position interventions.
Compare cached and uncached position assignments and logits.
Report training and evaluation lengths before making generalization claims.

Check Yourself

Can you distinguish permutation invariance from equivariance?
Can you explain why a fixed causal mask changes the reordering proof?
Can you calculate and count learned absolute position vectors?
Can you calculate four sinusoidal coordinates without memorizing a table?
Can you state a relative-offset sign convention and follow it consistently?
Can you rotate one RoPE pair and derive the relative-dot-product identity?
Can you separate an ALiBi bias from a causal mask?
Can you distinguish a computable index from demonstrated length generalization?
Can you locate a position bug that preserves every tensor shape?

Position information now reaches the attention calculation through a declared mechanism. Chapter 4 will place that attention update inside a complete Transformer block with residual paths, normalization, and an MLP.

References

Ashish Vaswani et al., Attention Is All You Need, 2017.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani, Self-Attention with Relative Position Representations, 2018.
Jianlin Su et al., RoFormer, 2021.
Ofir Press, Noah A. Smith, and Mike Lewis, Train Short, Test Long, 2022.
Adi Haviv et al., Transformer Language Models without Positional Encodings Still Learn Positional Information, 2022.