Compare and Audit Position Methods
Compare methods by where they enter the computation, which parameters they add, which indices they can represent, and what evidence supports longer contexts. Audit IDs, padding, masks, rotations, biases, and caches for silent errors.
Position methods differ first in where they enter the computation. A useful comparison starts there instead of asking which method is universally best.
| Method | Where position enters | Learned position entries | Pairwise distance enters directly? |
|---|---|---|---|
| Learned absolute | added to token representation | no | |
| Sinusoidal | added to token representation | 0 | indirectly through fixed frequency pairs |
| Shaw-style relative | score and optionally value interaction | relation-table dependent | yes |
| RoPE | rotations of query and key pairs | 0 in the original frequency schedule | yes, in query-key dot products |
| ALiBi | additive attention-score bias | 0 in the original fixed slope schedule | yes |
This table describes the versions taught in this chapter. Later variants may learn frequencies, rescale indices, bucket offsets, rotate only part of a head, or combine methods.
Four Questions for Every Method
1. What is the index convention?
State whether positions begin at 0, how left or right padding affects IDs, how special tokens are counted, and whether the key-minus-query or query-minus-key offset is used.
2. Where is the operation applied?
Adding a model-width vector, rotating Q/K pairs, adding a score bias, and altering values produce different functions. A matching final shape cannot prove that the operation was placed correctly.
3. What range can the implementation represent?
A learned table has a finite row range. A formula can calculate new indices, but numerical precision or a configured cache may impose other limits. This question concerns execution, not trained-model quality.
4. What evidence supports behavior outside the training range?
Report training length, evaluation lengths, model, data, metric, baselines, and any interpolation or scaling procedure. One perplexity comparison does not establish every long-context capability.
Interpolation and Extrapolation Are Different
- Interpolation evaluates positions within the range represented during training, even if a particular sequence length was not observed.
- Extrapolation evaluates positions or lengths beyond the training range.
Some context-extension methods rescale large inference indices into a smaller effective range. That is an interpolation strategy applied to position values, not unchanged extrapolation. Chapter 10 will compare such variants after the base mechanisms are established.
A Position Audit
For one fixed token batch, record:
- token IDs and padding locations;
- position IDs for every non-padding token;
- causal and padding masks separately;
- token and position vectors before addition, if used;
- unrotated and rotated Q/K pairs, if used;
- content scores and position biases separately;
- the combined score tensor before softmax;
- selected normalized rows and forbidden zeros;
- cached and uncached logits under identical position IDs;
- behavior at the largest trained and evaluated indices.
Valid Shapes Can Hide Position Errors
Common silent failures include:
| Failure | Why shape checks pass | Direct evidence |
|---|---|---|
| left-padded tokens receive shifted IDs at evaluation only | ID tensor retains | print token-to-position mapping |
| RoPE pairing convention changes | Q and K retain their widths | compare selected rotated coordinates and logits |
| key offset uses instead of | relation lookup still returns a vector | test asymmetric signed offsets |
| cached decode restarts the next token at 0 | cache and logits retain expected shapes | compare cached and full-prefix logits |
| ALiBi slopes attach to different heads | score tensor retains | log per-head bias rows |
Q1. Diagnose a cached position error
Full-prefix decoding assigns positions to four tokens. Cached decoding stores the first three keys correctly but assigns the fourth token position 0. Tensor shapes and cache length are correct, but logits differ. Which check most directly identifies the error?
Select one choice, then check.
Hint
Solution
Choose by Requirements and Evidence
A position method is part of an architecture contract. Choose or evaluate it using the required context, training setup, memory and parameter constraints, cache behavior, available evidence, and reproducibility of its conventions. Method names alone are not enough.