Compression as Prediction
Compression and prediction are closely related.
If you can predict the next symbol well, you can encode it with fewer bits on average. If you are often surprised, you need more bits.
Why This Matters
Language models learn to predict tokens.
Good token prediction means the model has learned useful structure in text: grammar, facts, recurring patterns, and styles.
This does not mean compression is the whole story of intelligence. It does mean prediction gives a powerful training signal.
Small Example
If a message is completely predictable, it can be described cheaply. If every symbol is surprising, it costs more to describe.
For example, if the next word is almost certainly "the", we do not need many bits to identify it. If many words are equally plausible, we need more bits.
In ML
This lens helps explain why next-token prediction is powerful. To predict well, a model must learn regularities in text. But compression alone does not tell us whether a system is truthful, aligned, or useful in every setting.
So compression is a lens, not a verdict. It helps explain why prediction learns structure, but it does not decide whether the learned structure should be trusted in a particular use.
Enter 1 if better prediction usually allows better compression.
Compute it first, then check your number.
Hint
Think of using fewer bits when the next symbol is expected.
Solution
Enter 1. Better prediction usually allows better compression because expected events require fewer bits to encode.
If a symbol is very surprising, does it usually require more bits to encode?
Answer it first, then check.
Hint
Surprise and code length move together.
Solution
Yes. Surprising events usually require more bits because the model did not expect them.
Does compression by itself fully explain every model capability?
Answer it first, then check.
Hint
The lesson calls compression a useful lens, not the whole story.
Solution
No. Compression is a useful lens for language modeling, but it is not a complete explanation of every model capability.
To predict tokens well, does a model need to learn regularities in text?
Answer it first, then check.
Hint
Good prediction depends on patterns in the data.
Solution
Yes. Good token prediction requires learning regularities such as grammar, facts, patterns, styles, and other structure in text.
Enter 1 if compression is a useful lens for language modeling but not a full
test of truthfulness or safety.
Compute it first, then check your number.
Hint
Ask whether predicting text well is the same as being truthful in every setting.
Solution
Enter 1. Compression explains why prediction can learn useful structure, but
truthfulness, safety, and usefulness require additional evaluation.
Before Moving On
Compression is a useful lens for language modeling, but it should not be treated as a complete explanation of every model capability.