Byte Pair Encoding

Byte pair encoding, or BPE, starts with small units and repeatedly merges the most frequent adjacent pair.

In text tokenization, a simple version works like this:

1. Start with characters.
2. Count adjacent pairs.
3. Merge the most frequent pair into a new token.
4. Repeat until the vocabulary reaches the chosen size.

The method is greedy. Each merge is chosen from the current corpus state.

Small example

Start with these words:

low
lower
lowest

As characters:

l o w
l o w e r
l o w e s t

The pair l o appears three times. A first merge could create lo:

lo w
lo w e r
lo w e s t

Now the pair lo w appears three times. Another merge could create low.

The tokenizer is learning reusable pieces from frequency.

Exercise

In the character view above, how many times does the adjacent pair l o occur?