Position Indices Must Continue through the Cache

Use cache length to assign the next absolute position, distinguish returned tokens from tokens already consumed as inputs, and stop before a learned position table is indexed beyond its context limit.

The cache length counts how many input positions the model has already processed. For learned absolute positions, the next input token uses exactly that index.

If the prompt length is $n$ , prefill uses position rows

P[0],P[1],\ldots,P[n-1].

The first token fed back during decode uses $P[n]$ , then $P[n+1]$ , until the context limit is reached.

Trace a Prompt Handoff

For prompt <bos> A:

Consumed token	Position row	Cache length after processing
`<bos>`	0	1
`A`	1	2
selected `B`	2	3
selected `D`	3	4

The model can then predict <eos> from the position-3 logits. It cannot feed that token at position 4 because the frozen table has rows 0 through 3 only.

Position Drift Produces Large Errors

The executable failure case assigns position 0 to every cached token. For prompt <bos> A, its maximum logit error is 23.0252 relative to the correct cached path. Shapes, cache lengths, and token IDs remain valid.

This is why cached-versus-full equivalence must test numerical values, not only cache structure.

Other Position Methods Keep the Same Principle

With RoPE, rotate each new query and key using the position of that token.
With ALiBi, calculate each new query-key distance using the continued indices.
With another relative method, preserve the same pairwise relation convention used during full-prefix execution.

The stored data may differ, but prefill and decode must represent the same positions.

Q1. Choose the next position

A prompt of length 3 has been prefilled into the cache. Which learned absolute position row should be added to the next token fed into the model?

Your answer

Answer it first, then check.

Hint

The next zero-based index equals the current cache length.

Solution

Use

P[3]

Position State Belongs to the Request

Do not derive the index from a global generation counter shared by several requests. Each sequence owns its prompt length, cache length, and stopping state.