Scripts Without Spaces
English often separates words with spaces. Not all writing systems do.
Some languages commonly write text without spaces between word-like units. Others use spacing and punctuation differently from English. A tokenizer that assumes spaces mark words will fail or behave poorly in those settings.
This chapter does not teach multilingual tokenization in depth. It only protects the main idea: word tokenization is not a universal starting point.
If a model is meant to work across languages, scripts, or mixed text, the text pipeline must be designed and tested with that reality in mind.
Enter 1 if every writing system reliably separates words with spaces, or 0 if that assumption fails.
Compute it first, then check your number.
Subword and byte-level methods partly respond to these difficulties, but they do not remove the need to inspect text carefully.