Categorical and Multinomial Distributions

Model one labeled outcome with a categorical variable and repeated category counts with a multinomial vector. Use one-hot representations, distinguish sequence from count probability, compute count and proportion moments, and state what order and dependence the model discards.

Categorical and multinomial distributions describe trials with a finite set of possible categories, commonly more than two. A categorical variable records one category. A multinomial variable records how many times each category appears across a fixed number of independent trials with unchanged category probabilities.

One Outcome from Several Categories

Let a variable $Y$ take one of $K$ labels with probability vector:

\mathbf{p}=(p_1,\ldots,p_K), \qquad p_k\ge0, \qquad \sum_{k=1}^{K}p_k=1.

Then $Y$ has a categorical distribution. For three weather labels, one possible model is:

P(Y=k)=p_k, \qquad k\in\{1,\ldots,K\}.

Category	sunny	cloudy	rainy
Probability	$0.50$	$0.30$	$0.20$

The labels identify alternatives; they need not carry numerical distance or order. Renaming sunny, cloudy, and rainy as $1$ , $2$ , and $3$ does not make “rainy minus cloudy” a meaningful quantity.

A Bernoulli distribution is the two-category case when the labels are encoded as zero and one.

Exercise: Complete a categorical distribution

A three-class distribution assigns probabilities $0.5$ , $0.3$ , and $p_3$ . What is $p_3$ ?

Check the missing mass

Compute it first, then check your number.

HintNormalize the category probabilities

Subtract the two known probabilities from one.

SolutionAssign all remaining probability mass

A categorical probability vector sums to one:

p_3=1-0.5-0.3=0.2.

The result is nonnegative, so the completed vector is valid.

One-Hot Vectors Represent Categories Numerically

A $K$ -category outcome is often represented by a one-hot vector $\mathbf{Y}\in\mathbb{R}^K$ . Exactly one component equals one and the others equal zero. For the three weather categories:

\text{sunny}=(1,0,0),\quad \text{cloudy}=(0,1,0),\quad \text{rainy}=(0,0,1).

The expected one-hot vector is the probability vector itself:

\mathbb{E}[\mathbf{Y}]=\mathbf{p}.

This representation appears in classification. A model produces a probability vector over classes, while the observed class can be encoded as one-hot. A language model's next-token distribution is categorical over its vocabulary.

Exercise: Interpret the expected one-hot vector

For category probabilities $(0.2,0.5,0.3)$ , what is the second component of $\mathbb{E}[\mathbf{Y}]$ ?

Check the expected component

Compute it first, then check your number.

HintAverage each indicator component

The second component equals one exactly when category two occurs.

SolutionAn indicator's expectation is its event probability

The second one-hot component is an indicator for category two. Its expected value is therefore $P(Y=2)=0.5$ .

Repeated Categories Produce a Count Vector

Repeat the categorical trial $n$ times independently with the same probability vector $\mathbf p$ . Let $N_k$ count occurrences of category $k$ . The count vector satisfies:

N_1+\cdots+N_K=n.

It has a multinomial distribution:

P(N_1=n_1,\ldots,N_K=n_K) =\frac{n!}{n_1!\cdots n_K!} \prod_{k=1}^{K}p_k^{n_k}.

“The probability of the count vector equals the multinomial coefficient times the product, over categories $k$ , of $p_k$ raised to count $n_k$ .” The coefficient counts the different orders represented by the same counts.

This formula applies to nonnegative counts whose sum is $n$ . The product gives the probability of one particular sequence with those counts. The multinomial coefficient counts how many distinct sequences have the same count vector. As usual, $0!=1$ , so a category with zero observations does not alter the coefficient or product.

For $n=3$ , probabilities $(0.5,0.3,0.2)$ , and counts $(2,1,0)$ :

P(2,1,0) =\frac{3!}{2!1!0!}(0.5)^2(0.3)^1(0.2)^0 =3(0.25)(0.3) =0.225.

The three matching sequences are first-first-second, first-second-first, and second-first-first.

Exercise: Compute a multinomial count probability

Two independent trials have category probabilities $(0.5,0.3,0.2)$ . What is the probability that category one occurs once, category two occurs once, and category three does not occur?

Check the count probability

Compute it first, then check your number.

HintCount the two possible orders

The matching sequences have probabilities $0.5(0.3)$ and $0.3(0.5)$ .

SolutionAdd both arrangements of the counts

The multinomial coefficient is $2!/(1!1!0!)=2$ , so:

P(1,1,0)=2(0.5)(0.3)=0.3.

Counting only one order would miss half of the required event.

A Sequence and Its Count Vector Are Different Events

For probabilities $(0.5,0.3,0.2)$ , the exact ordered sequence first-first-second has probability:

(0.5)(0.5)(0.3)=0.075.

The count event $(N_1,N_2,N_3)=(2,1,0)$ includes three ordered sequences, so its probability is $3(0.075)=0.225$ . The multinomial coefficient appears only because the count vector groups several sequences together.

This distinction matters in language. A token sequence model assigns probability to an order. A multinomial bag-of-words model assigns probability to counts and deliberately discards order.

Exercise: Separate sequence probability from count probability

Three independent categorical trials use probabilities $(0.6,0.4)$ . What is the probability of the exact ordered sequence first-second-first?

Check the sequence probability

Compute it first, then check your number.

HintDo not add other orders

Multiply the probabilities in the specified order: $0.6(0.4)(0.6)$ .

SolutionMultiply one specified sequence

The exact sequence has probability:

(0.6)(0.4)(0.6)=0.144.

No combinatorial coefficient is used because the prompt names one order, rather than every sequence containing two first-category outcomes.

Means, Variances, and Dependence among Counts

Each individual count has mean and variance:

\mathbb{E}[N_k]=np_k, \qquad \operatorname{Var}(N_k)=np_k(1-p_k).

Increasing $p_k$ moves more expected count toward category $k$ because $\mathbb{E}[N_k]=np_k$ . The probabilities must still sum to one, so increasing one category's probability requires decreasing one or more of the others. Increasing $n$ scales every expected count in proportion to the number of trials, while the observed proportions $N_k/n$ become more stable around the probability vector as the sample grows.

Viewed alone, $N_k$ has a binomial distribution. However, the counts are not independent because their sum is fixed. For $j\ne k$ :

\operatorname{Cov}(N_j,N_k)=-np_jp_k.

An extra occurrence in one category leaves one fewer trial available for the others, producing negative covariance.

Exercise: Compute an expected category count

For $n=100$ trials and category probabilities $(0.6,0.25,0.15)$ , what is $\mathbb{E}[N_2]$ ?

Check the expected count

Compute it first, then check your number.

HintMultiply total trials by the category probability

Use $np_2$ .

SolutionAdd the expected indicators for category two

The expected count is:

\mathbb{E}[N_2]=100(0.25)=25.

Other count values remain possible; $25$ is the distribution's mean for that component.

Count Proportions Estimate Category Probabilities

Define the empirical category proportion:

\widehat p_k=\frac{N_k}{n}.

Scaling the count moments gives:

\mathbb{E}[\widehat p_k]=p_k, \qquad \operatorname{Var}(\widehat p_k)=\frac{p_k(1-p_k)}{n}.

The proportions always satisfy $\sum_k\widehat p_k=1$ . Their negative covariances also shrink with sample size:

\operatorname{Cov}(\widehat p_j,\widehat p_k) =-\frac{p_jp_k}{n}, \qquad j\ne k.

As $n$ grows, empirical proportions tend to concentrate around the probability vector, while individual samples still fluctuate.

Exercise: Compute the variance of a category proportion

For $n=200$ independent categorical trials with $p_1=0.5$ , what is $\operatorname{Var}(\widehat p_1)$ ?

Check the variance

Compute it first, then check your number.

HintScale the indicator variance by sample size

Use $p_1(1-p_1)/n$ .

SolutionMeasure the spread of an empirical proportion

The variance is:

\operatorname{Var}(\widehat p_1) =\frac{0.5(0.5)}{200} =0.00125.

Increasing the number of trials reduces this variance because the empirical proportion averages more categorical indicators.

Choose the Distribution by the Recorded Object

The same trial can lead to different random variables:

Recorded object	Distribution
one binary outcome	Bernoulli
number of successes in repeated binary trials	Binomial
one outcome among $K$ categories	Categorical
vector of category counts across repeated trials	Multinomial

If order is retained, a full sequence of categorical variables is being modeled; the multinomial count vector discards that order. Two documents with the same token counts but different token sequences have the same count vector, so a multinomial bag-of-words model cannot distinguish them.

Exercise: Choose the matching distribution

A document contains $100$ independently sampled tokens from a three-token vocabulary. Only the count of each token is recorded. Which distribution models the three counts under these assumptions?

Choose the distribution

Bernoulli

Categorical

Multinomial

Select one choice, then check.

HintIdentify what was repeated and retained

There are many multi-category trials, and the recorded object is a count vector.

SolutionMatch the multinomial count vector

One token draw is categorical. Repeating the draw $100$ times and retaining one count per category gives a multinomial distribution, provided the draws are independent and share one fixed probability vector.

Check the Repeated-Trial Assumptions

The multinomial model requires a fixed number of independent trials and one unchanged categorical probability vector. Natural language tokens generally do not satisfy this assumption: the next-token distribution changes with context, and nearby tokens are dependent. A multinomial token-count model can still be a deliberate approximation, but its limitations should be explicit.

Simulate Categorical Draws and Count Them

Compare multinomial counts with expected counts

Change the category probabilities, trial count, or seed. Compare one simulated count vector with its expectation.

Ready to run.

One sample need not equal its expected count vector. Repeating the complete experiment and averaging its counts would approach the expectation.

Before Choosing a Multi-Category Distribution

Use categorical for one labeled outcome and multinomial for a vector of counts from repeated independent categorical trials with fixed probabilities. Decide whether order has been discarded, verify normalization, and state whether the independence and constant-probability assumptions are exact or approximate.

The next lessons turn to continuous distributions, where probability belongs to intervals and regions rather than individual points.