Nucleus Sampling Preview

Nucleus sampling, also called top-p sampling, keeps the smallest set of tokens whose total probability reaches a threshold p.

If p = 0.90, the decoder includes the most likely tokens until their cumulative probability is at least 0.90, then samples from that set.

Unlike top-k, the number of kept tokens can change from step to step. A sharp distribution may keep few tokens. A flat distribution may keep more.

Exercise

If top-p uses p = 0.90, what threshold percentage is being targeted?