Distributions
A distribution describes how probability is assigned to possible values.
For a discrete random variable, the probabilities can be listed.
Example
Suppose a classifier predicts one of three classes:
cat: 0.70
dog: 0.20
car: 0.10
These probabilities form a distribution over classes.
They should sum to 1.
The distribution answers questions such as:
- What value is most likely?
- Which values are possible?
- How much probability is assigned to a group of values?
For the classifier above, the event {cat, dog} has probability:
Discrete and Continuous
A discrete distribution assigns probability to separate values.
A continuous distribution assigns probability over intervals.
For continuous variables, the probability of one exact value is usually not the main object. We ask for probability over a range.
This chapter mostly uses discrete examples so the arithmetic is visible. The same habit carries forward: define possible values, assign probability consistently, and ask what the distribution says.
ML Reading
Language models output a distribution over the next token.
Classification models output a distribution over classes.
Generative models define or learn distributions over data.
In all three cases, a distribution is not only a list of numbers. It is a claim about uncertainty under a model. The same probabilities should be interpreted relative to the model, the input, and the set of possible outcomes.
A model outputs probabilities [0.2, 0.3, 0.5].
What is their sum?
Compute it first, then check your number.
Hint
Add the three probabilities.
Solution
0.2 + 0.3 + 0.5 = 1. For exclusive outcomes, the probability mass across all
listed possibilities must sum to one.
A classifier outputs:
cat: 0.70
dog: 0.20
car: 0.10
Which label is most likely?
Answer it first, then check.
Hint
Choose the label with the largest probability.
Solution
cat has probability 0.70, which is the largest. The most likely label is
the one with the greatest assigned probability.
Can [0.4, 0.4, 0.4] be a valid distribution over three exclusive outcomes?
Answer it first, then check.
Hint
Add the probabilities.
Solution
No. The probabilities sum to 1.2, not 1. That assigns more than total
certainty across the three exclusive outcomes.
Is a distribution over labels {cat, dog, car} discrete or continuous?
Answer it first, then check.
Hint
The possible values are separate labels.
Solution
It is discrete because the possible values are separate labels. Probability is assigned to distinct choices rather than intervals on a continuum.
Enter 1 if a language model's next-token probabilities are a distribution
conditioned on the current context.
Compute it first, then check your number.
Hint
Ask what the probabilities are conditioned on.
Solution
Enter 1. A next-token distribution assigns probability to possible next
tokens given the current context.
Before Moving On
A distribution is a map from possible values to probability.