Punctuation and Whitespace

Whitespace is not always a single ordinary space. Text may contain tabs, new lines, non-breaking spaces, or repeated spaces.

Punctuation also carries information:

go
go!
go?

These are not the same message. Removing punctuation can simplify a toy count model, but it can erase tone, sentence boundaries, questions, code syntax, and format.

For language modeling, punctuation and whitespace often become tokens or affect tokens. That is why modern tokenizers usually preserve more structure than a simple split(" ") approach.