Special Tokens, Padding, and Truncation

A tokenizer usually has tokens that are not ordinary text pieces.

Common special tokens include:

• a beginning-of-sequence token;
• an end-of-sequence token;
• a padding token;
• an unknown token;
• separator or mask tokens in some model families.

These tokens are part of the contract between the tokenizer and model.

Padding

Neural networks often process batches. If one sequence has 5 tokens and another has 9, padding can make them the same length:

real real real real real pad pad pad pad
real real real real real real real real real

The model or loss must know which positions are padding.

Truncation

Models have a maximum context length. If a sequence is too long, the system may cut it. That choice can remove important evidence.

Padding and truncation are not cosmetic details. They decide what information reaches the model.

Exercise

A batch contains sequences of length 4 and 7. If both are padded to length 7, how many padding tokens are added to the shorter sequence?