ALiBi Adds Distance Bias to Scores

Attention with Linear Biases, or ALiBi, does not add a position vector to each token. It adds a head-specific distance penalty to attention scores before softmax.

For causal attention using query index $i$ and readable key index $j\le i$, one convention is

where $m_r>0$ is the slope for head $r$. Earlier keys have larger distances and receive more negative biases.

Calculate One Biased Row

Let query index $i=3$ read key indices $j=0,1,2,3$. Suppose the content scores are all zero and the head slope is $m_r=0.5$. The distances are

so the biases are

After softmax, the attention weights are approximately

The closest key receives the largest weight only because the content scores were equal. A sufficiently strong content score for an earlier token can overcome the distance penalty.

Different Slopes Give Different Distance Preferences

A small slope changes slowly with distance. A larger slope more strongly prefers recent keys, all else equal. Assigning different slopes to different heads gives the layer several fixed distance priors without learning a position embedding table.

The slopes are part of the method's configuration. Their schedule must be reproduced when porting a model. Replacing every slope with one shared value changes the model even though all tensor shapes remain valid.

Bias Is Not a Mask

A causal mask assigns forbidden future keys an effectively negative-infinite score, making their weights zero. ALiBi applies finite penalties to allowed keys. It shapes preference but does not change which positions are legal. Implementations may combine the causal mask and ALiBi bias in one additive tensor, but their meanings remain different.

Keep the Extrapolation Claim Bounded

Press, Smith, and Lewis trained a 1.3-billion-parameter language model on length-1024 inputs and evaluated it at length 2048. In their reported setup, ALiBi matched the perplexity of a sinusoidal model trained at length 2048 while reducing training memory and time under the stated comparison.

This evidence motivates the method. It does not guarantee the same result for every architecture, dataset, task, context extension, slope schedule, or evaluation metric.

Reference

• Ofir Press, Noah A. Smith, and Mike Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, ICLR 2022.