What Content-Only Attention Preserves Under Reordering

Unmasked content-only attention follows a joint row permutation rather than ignoring output order. Derive this equivariance, then show why a fixed causal mask breaks arbitrary permutation symmetry and can supply an implicit position signal.

Before adding position information, we need a precise account of what attention already does with row order. The result depends on whether attention is unmasked or uses a position-dependent mask.

Equivariance Is Not Invariance

Let $P$ be a permutation matrix that reorders the $T$ rows of $X$ . For example, $P$ may change the row order from $[0,1,2]$ to $[2,0,1]$ .

An invariant function would return the same output after the reordering:

f(PX)=f(X).

An equivariant function reorders its output in the corresponding way:

f(PX)=Pf(X).

Self-attention returns one record per input position, so equivariance—not invariance—is the relevant property.

Derive the Unmasked Result

For one unmasked head,

Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.

After reordering $X'=PX$ , the shared projections give

Q'=PQ,\qquad K'=PK,\qquad V'=PV.

Let $S=QK^T/\sqrt{d_k}$ . After reordering, the score matrix becomes

S'=\frac{Q'K'^T}{\sqrt{d_k}} =\frac{(PQ)(PK)^T}{\sqrt{d_k}} =PSP^T.

Row-wise softmax follows the same row-and-column permutation:

A'=PAP^T.

Therefore,

Z'=A'V'=(PAP^T)(PV)=PA(P^TP)V=PZ.

The reordered operation follows the records, but it has not learned a separate meaning for index 0 or index 2.

A Fixed Causal Mask Changes the Argument

Causal attention adds a triangular mask $M$ before softmax:

A=\operatorname{softmax}_{last}(S+M).

If we reorder only the input while keeping the usual mask tied to indices $0,1,\ldots,T-1$ , then in general

P(S+M)P^T \ne PSP^T+M.

The first expression also permutes the mask. The second keeps “may read only this index and earlier indices” attached to the new row numbers. An arbitrary reordering therefore changes which tokens may interact, not only where the same result appears.

This is one reason causal language models can obtain some position signal even without an explicit position vector: successive query indices have different visible prefixes. The signal is implicit in the computation structure. It is not the same as adding a learned absolute vector or changing scores according to a relative distance.

Verify Both Cases

Reorder unmasked and causal attention

The program holds projection matrices fixed, reorders the input, and checks the equivariance relation with and without a fixed causal mask.

The first comparison should be True. The second should be False. This is not a software defect: the fixed triangular mask gives the row indices a directional visibility structure.

No Explicit Encoding Does Not Mean No Position Signal

Haviv and colleagues trained causal Transformer language models without an explicit positional encoding and found that the models still learned position-related information and were competitive in their tested settings. This result corrects an overgeneralization; it does not prove that every task, architecture, context length, or training setup can omit explicit position methods without cost.

Q1. Distinguish the two reordering claims

An unmasked content-only self-attention layer receives $PX$ instead of $X$ . What output relation should hold when all projections are shared by position?

Choose one

Attention(PX) = P Attention(X)

Attention(PX) = Attention(X)

Attention(PX) always equals P Attention(X), even with a fixed triangular mask

Select one choice, then check.

Hint

Use the distinction between an unchanged result and a correspondingly reordered result.

Solution

The equivariant relation holds. With a fixed causal mask, an arbitrary permutation generally changes visibility and invalidates this simple relation.

State the Setting Before the Claim

When discussing position, first name the attention setting: unmasked, causal, padding-masked, or another structured mask. Then name whether position is explicit in input vectors, query-key transforms, scores, values, or only the visibility pattern. This habit prevents a correct statement about one setting from becoming a false statement about every Transformer.

References

Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy, Transformer Language Models without Positional Encodings Still Learn Positional Information, Findings of EMNLP 2022.