Merge Operations

A BPE tokenizer is defined by an ordered list of merges. The order matters. Later merges can only apply after earlier merges have changed the text.

Suppose the merge list is:

l + o -> lo
lo + w -> low
e + r -> er

Apply it to:

l o w e r

After the first merge:

lo w e r

After the second merge:

low e r

After the third merge:

low er

The final tokenization is low er.

Why this matters

The model never sees the original characters directly unless the tokenizer leaves them as tokens. The merge list shapes the sequence length, vocabulary ids, and what counts as a reusable pattern.

Exercise

Using the merge list above, how many tokens remain after tokenizing lower?