Decoding Policy Acts on Fixed Logits

Hold one logits row fixed while comparing greedy decoding, temperature, top-k restriction, and seeded sampling. Learn which changes affect the probability distribution and which affect only selection.

The model returns logits. A decoding policy transforms or filters that row and then selects a token. Keep the original logits fixed when comparing policies.

Greedy Choice

Greedy decoding uses

\hat y=\arg\max_i z_i.

It is deterministic for fixed logits and a fixed tie rule. It does not return the most probable complete sequence in general; it makes one locally largest choice at each step.

Temperature

For temperature $\tau>0$ :

p_i=\frac{\exp(z_i/\tau)}{\sum_j\exp(z_j/\tau)}.

$\tau<1$ sharpens differences;
$\tau>1$ flattens them;
$\tau=1$ leaves the original softmax distribution;
this chapter handles $\tau=0$ as an explicit greedy request;
negative temperature is rejected.

For logits $[2,1,0]$ , temperature 0.5 gives scaled logits $[4,2,0]$ and probabilities approximately $[0.8668,0.1173,0.0159]$ .

Top- $k$ Filtering

Top- $k$ keeps the $k$ largest logits, sets the rest to $-\infty$ , and renormalizes.

$k=1$ becomes greedy after tie handling;
$k=V$ removes no token;
$k<1$ or $k>V$ is invalid.

Filtering changes the sampling distribution. It does not change model parameters or the unfiltered logits.

Sampling Needs Its Own Random State

The verified sampled run uses seed 41, temperature 1, and top- $k=3$ :

<bos> Y X Y <eos>

A different seed can produce another valid draw from the same probabilities. One attractive sample is not evidence of lower loss or better model quality.

Q1. Scale logits by temperature

What logits result from dividing [3,1,-1] by temperature 2?

Your answer

Compute it first, then check your number.

Hint

Calculate each coordinate independently.

Solution

The scaled logits are [1.5,0.5,-0.5].

Policy Is Not a Cache Setting

Cached and uncached execution should produce the same logits. Apply the same policy and random-generator state only after that equality is established.