LLM Primer

Introduction

Word tokenization makes sequences short, but it cannot handle every new word. Character tokenization handles new spelling, but it makes sequences long. Subword tokenization is the compromise.

A subword tokenizer can split unhelpful into pieces such as un, help, and ful. The model does not need a separate vocabulary entry for every full word, but it also does not need to read one character at a time.

This chapter introduces subword tokenization as a practical design before we study probability models. The goal is not to memorize tokenizer brands. The goal is to see the tradeoff: a tokenizer chooses what the model is allowed to see and predict.

BPE builds larger pieces by repeated mergeslowermerge l + olowereach merge changes the sequence that later merges will see
BPE starts with small units and creates reusable subword pieces from frequent adjacent pairs.

What this chapter covers

  • why rare words create an open-vocabulary problem;
  • how byte pair encoding builds larger pieces by repeated merges;
  • how merge operations change the same text over time;
  • how WordPiece and unigram tokenization fit the same family of ideas;
  • why special tokens, padding, and truncation are part of the model interface.