WordPiece and Unigram Tokenization

BPE is not the only subword method. Two other names often appear around modern language models: WordPiece and unigram tokenization.

You do not need their full training algorithms yet. For now, keep the shared idea in view: all three methods build a fixed vocabulary of reusable pieces. They differ in how they choose those pieces and how they score a segmentation.

WordPiece

WordPiece also builds subword units, but it chooses merges using a likelihood or score criterion rather than the plain most-frequent-pair rule used in the simple BPE story.

The important practical point is the same: common chunks get their own tokens; rare words can be decomposed.

Unigram tokenization

Unigram tokenization starts with many candidate pieces and learns which pieces are useful. It can choose a likely segmentation from several possible segmentations.

That makes tokenization feel less like a single greedy merge list and more like a small probabilistic model over pieces.

Exercise

How many of these are subword-tokenization families: BPE, WordPiece, unigram?