UTF-8

UTF-8 is a common way to encode Unicode text as bytes.

Plain ASCII characters use one byte in UTF-8. Many other characters use more than one byte.

That means a short visible string can take different amounts of storage:

A      -> often 1 byte in UTF-8
é      -> often 2 bytes in UTF-8
न      -> often 3 bytes in UTF-8

You do not need to memorize byte patterns here. You need the principle: bytes are not the same as visible characters.

Byte-level tokenizers use this fact deliberately. They can represent any text as bytes, which helps avoid unknown characters, but the token sequence may no longer match what a reader thinks of as characters or words.

LM-C03-T02-001Exercise: ASCII byte count

In UTF-8, plain ASCII cat uses 1 byte per letter. How many bytes does cat use?

Compute it first, then check your number.