Batches, Stopping, and Context Limits Complicate Decode

A single sequence has one cache length and one stopping state. A batch may contain different prompt lengths, choose <eos> at different steps, or reach context limits at different times.

Keep State per Sequence

For each batch row, track:

• prompt and generated IDs;
• valid cached length;
• position index for the next consumed token;
• active or finished state;
• stopping reason;
• decoding-policy random state where independent streams are required.

Do not let one finished row append ordinary tokens merely because another row is still active.

Common Handling Strategies

An implementation can:

• pad cache tensors and use validity masks;
• group sequences with similar lengths;
• remove finished rows from an active batch and remap state;
• use a runtime that supports variable-length cache blocks.

Each strategy changes bookkeeping and performance, not the model's intended logits for a given valid prefix.

Context Limit and EOS Are Different

• eos means the policy selected a learned stopping token.
• context means the model has no valid next input position under this configuration.

A sequence can reach the context limit without selecting <eos>. Report that distinction to the caller.

Padding Must Not Become Evidence

If cache tensors are padded to a common length, prevent queries from reading unused cache slots. A causal relation alone does not identify which earlier slots are padding in another batch row.

Serving Systems Are a Later Layer

Continuous batching, cache paging, scheduling, request admission, and memory reclamation deserve a separate engineering treatment. This lesson establishes the state they must preserve.