LLM Primer

Introduction

A language model returns probabilities. Decoding turns those probabilities into generated text.

This choice matters. The same model can behave differently under greedy decoding, temperature sampling, top-k sampling, or nucleus sampling. The model has not changed. The policy for choosing tokens has changed.

decoding turns probabilities into chosen tokenstea0.45code0.30music0.15books0.10greedy, temperature, top-k, and nucleus sampling read the same distribution differently
The model gives probabilities; the decoding policy decides how text is actually generated.

What this chapter covers

  • greedy decoding;
  • temperature;
  • top-k sampling;
  • nucleus sampling preview;
  • repetition and uncertainty.