Generation Repeats Prediction and Appending

Trace autoregressive generation as repeated next-token prediction, selection, append, and stopping. Separate the model's logits from the decoding policy and record why each sequence stops.

For prompt IDs $[s_0,\ldots,s_{P-1}]$ , the decoder returns logits at every prompt position. Generation uses only the final row because that row predicts the token after the complete known prefix.

One step is:

z_{P-1}=f_\theta(s_0,\ldots,s_{P-1})[-1], \qquad s_P=\operatorname{choose}(z_{P-1}).

Append $s_P$ and repeat unless a stopping rule applies.

Trace the Frozen Greedy Run

Greedy decoding selects the largest logit. From <bos>, the verified model produces:

Known input	Selected next token	Action
`<bos>`	`A`	append
`<bos> A`	`B`	append
`<bos> A B`	`D`	append
`<bos> A B D`	`<eos>`	stop

The final sequence is <bos> A B D <eos>. The choice of D rather than C reflects the trained parameters and sampled corpus counts; both endings belong to the controlled grammar.

Stop before Creating an Invalid Input

Check in this order after choosing a token:

append the selected token to the returned sequence;
if it is <eos>, stop;
if the cache already holds $C$ input positions, stop with a context-limit reason rather than feeding another token;
otherwise process the selected token as the next input.

The model can predict one token beyond its last valid input position. It cannot calculate logits from that predicted token without a position row for it.

Generation Does Not Train the Model

Use evaluation and inference mode. Do not call backward or an optimizer. A KV cache assumes parameters remain unchanged; updating the model would make stored keys and values stale.

Q1. Trace one append step

The prompt is [<bos>, A], and the final logit row has its maximum at token B. What becomes the next model input if neither stopping rule applies?

Your answer

Answer it first, then check.

Hint

Do not replace the last prompt token.

Solution

The next input is [<bos>, A, B].

Record Why a Run Stopped

Store eos, context, or another explicit reason. Returning a short sequence without its stopping cause makes successful completion indistinguishable from a length or runtime failure.