Classification and Language Modeling
Information theory appears in two everyday ML settings: classification and language modeling.
Classification
In classification, the model predicts a probability distribution over classes.
Cross-entropy punishes the model when it assigns low probability to the correct class.
For an image classifier, the classes might be cat, dog, and car. For a
medical classifier, the classes might be possible diagnoses. The loss reads the
probability assigned to the observed label.
Language Modeling
In language modeling, the model predicts a probability distribution over the next token.
The same idea applies: assign high probability to the actual next token.
The vocabulary may contain thousands of tokens, but the pattern is still classification: choose a distribution over possible next tokens, then compare it with the token that actually appeared.
Shared Pattern
Both settings use:
- a target outcome
- a predicted distribution
- a loss based on probability assigned to the target
The difference is scale and context. Classification may use one label for one input. Language modeling repeats the same prediction problem at many positions in a sequence.
This is why next-token prediction can look simple and still be powerful. The single step is ordinary classification. The strength comes from repeating it over vast text, with rich context before each target.
Enter 1 if cross-entropy can be used for next-token prediction.
Compute it first, then check your number.
Hint
Next-token prediction is classification over a vocabulary.
Solution
Enter 1. Next-token prediction is a classification problem over possible tokens, so cross-entropy is a natural training loss.
In next-token prediction, are the possible tokens like classes for that prediction step?
Answer it first, then check.
Hint
The model predicts a distribution over possible next tokens.
Solution
Yes. For a single next-token prediction, the vocabulary tokens behave like the possible classes.
What does cross-entropy read from the model output: the probability assigned to the observed target, or only the target name?
Answer it first, then check.
Hint
The loss is based on probability assigned to the target.
Solution
The target name tells us which class was observed. Cross-entropy then reads the model probability assigned to that observed class. A correct label with low assigned probability still gives a high loss.
Does language modeling repeat the prediction problem across many positions in a sequence?
Answer it first, then check.
Hint
Each position has a next-token prediction target.
Solution
Yes. Language modeling repeats next-token prediction across many positions in a sequence.
Enter 1 if next-token prediction is a simple classification step repeated
across many contexts and positions.
Compute it first, then check your number.
Hint
Separate one prediction step from the amount of data and context used.
Solution
Enter 1. At each position, the model predicts a distribution over possible
next tokens. Language modeling repeats that classification problem many times.
Before Moving On
The details differ, but the core loss pattern is the same: probability assigned to the observed target.