Bits per Token and Byte

Cross-entropy can be reported in bits per token. But tokenizers differ, so token counts differ too.

One tokenizer may split a sentence into 8 tokens. Another may split the same sentence into 11 tokens. Their bits-per-token values are not directly comparable without care.

Bits per byte is sometimes used to reduce tokenizer dependence. It measures the average number of bits needed per byte of text.

Practical reading

When you see a language-model score, ask:

• what dataset was used?
• what tokenizer was used?
• was the score per token, byte, character, or word?
• was there any leakage?

The number means little without the evaluation setup.

Exercise

If a text has 20 bytes and total loss is 40 bits, what is the bits-per-byte value?