Top-k Sampling

Top-k sampling keeps only the k highest-probability tokens, renormalizes their probabilities, and samples from that smaller set.

If k = 3, only the top three candidates remain. Lower-probability candidates outside that set cannot be sampled.

Top-k is a way to avoid very unlikely tokens while still allowing variation.

Exercise

If k = 5, how many candidate tokens are kept before sampling?