Joint, Marginal, and Conditional Distributions

Keep several random variables together with a joint distribution, remove variables by marginalizing, and restrict known information by conditioning. Use the product rule and total probability to move between these views.

A joint distribution describes several random variables within the same experiment. From it, we can ignore a variable by marginalizing or use a known value by conditioning. These operations answer different questions and retain different amounts of information about how the variables vary together.

Build a Joint Distribution from Paired Observations

Suppose a weather station records two variables each morning:

$R=1$ if it rains and $R=0$ otherwise;
$W=1$ if the wind is strong and $W=0$ otherwise.

Across 100 mornings, the paired counts are:

	$W=0$	$W=1$	Row total
$R=0$	50	20	70
$R=1$	10	20	30
Column total	60	40	100

Dividing every count by $100$ gives the joint probability mass function:

	$W=0$	$W=1$	Row total
$R=0$	$0.50$	$0.20$	$0.70$
$R=1$	$0.10$	$0.20$	$0.30$
Column total	$0.60$	$0.40$	$1.00$

For discrete random variables $R$ and $W$ , write:

p_{R,W}(r,w)=P(R=r,W=w).

Each cell assigns probability to one ordered pair. For example, $p_{R,W}(1,0)=0.10$ is the probability of rain without strong wind. A valid joint mass function is nonnegative and sums to one over every possible pair:

\sum_r\sum_w p_{R,W}(r,w)=1.

The order of the variables must remain clear. The pair $(R,W)=(1,0)$ is not the same label as $(R,W)=(0,1)$ .

Exercise: Read a joint probability from a table

Using the weather table, what is $P(R=0,W=1)$ ?

Check the joint probability

Compute it first, then check your number.

HintFind one cell, not a row or column total

Locate the row $R=0$ and the column $W=1$ .

SolutionRead the probability assigned to the ordered pair

The corresponding cell is $0.20$ , so:

P(R=0,W=1)=0.20.

This cell represents mornings with strong wind and no rain.

Marginalization Removes a Variable

Suppose we want the distribution of rain without keeping track of wind. Add across every possible wind value:

p_R(r)=\sum_w p_{R,W}(r,w).

“The marginal mass at $r$ equals the sum of the joint masses at $(r,w)$ over every possible $w$ .” The sum removes $W$ while retaining $R$ .

For rainy mornings,

P(R=1)=P(R=1,W=0)+P(R=1,W=1)=0.10+0.20=0.30.

This is the marginal distribution of $R$ . The name comes from the row and column totals traditionally written in the margins of a table. We say that $W$ has been summed out.

Marginalization preserves the total probability for each remaining value but discards how that value was divided among values of the removed variable. From $P(R=1)=0.30$ alone, we cannot recover how many rainy mornings were windy.

Exercise: Sum out one variable

Using the weather table, what is the marginal probability $P(W=0)$ ?

Check the marginal probability

Compute it first, then check your number.

HintAdd over every rain value

Add the two cells in the $W=0$ column.

SolutionAdd the column to remove rain status

Sum over $R=0$ and $R=1$ :

P(W=0)=0.50+0.10=0.60.

The marginal retains whether wind is strong but no longer records whether it rained.

Conditioning Keeps a Slice and Renormalizes It

Now suppose we know that it rained. The row $R=1$ contains the relevant joint probabilities, but its total is $0.30$ , not one. Divide each cell in that row by the row total:

P(W=w\mid R=1) =\frac{P(W=w,R=1)}{P(R=1)}.

Therefore:

P(W=0\mid R=1)=\frac{0.10}{0.30}=\frac13, \qquad P(W=1\mid R=1)=\frac{0.20}{0.30}=\frac23.

The two conditional probabilities sum to one. Marginalization removed $R$ by adding over it; conditioning fixes $R$ at one value and rescales the selected slice.

Exercise: Normalize a selected table slice

Using the weather table, calculate $P(R=1\mid W=1)$ .

Check the conditional probability

Compute it first, then check your number.

HintUse the strong-wind column as the conditioned group

The joint cell is $0.20$ , and the $W=1$ column total is $0.40$ .

SolutionDivide the selected cell by its column total

Restrict attention to mornings with strong wind:

P(R=1\mid W=1)=\frac{P(R=1,W=1)}{P(W=1)} =\frac{0.20}{0.40}=0.50.

Half of the strongly windy mornings are rainy in this model.

A Joint Distribution Retains Dependence

The two marginals do not generally determine the joint distribution. In the weather model:

P(R=1)=0.30, \qquad P(W=1)=0.40.

If rain and strong wind were independent, their joint probability would be:

P(R=1)P(W=1)=0.30(0.40)=0.12.

The actual joint probability is $0.20$ . The difference records an association that cannot be seen from either marginal alone. Many distinct joint tables can have the same row and column totals while arranging probability differently inside the table.

Exercise: Detect information lost by marginalization

Suppose you know only that $P(R=1)=0.30$ and $P(W=1)=0.40$ . Can you determine $P(R=1,W=1)$ without an independence assumption or other joint information?

Choose one

Yes; it must equal $0.30(0.40)$ .

No; the two marginals do not determine their joint arrangement.

Select one choice, then check.

HintAsk whether multiplication is always valid

The product $P(R=1)P(W=1)$ equals the joint probability only under an independence condition.

SolutionRetain the missing dependence information

No. The marginals specify the row and column totals, but probability can be arranged inside those totals in different ways. An independence assumption would select $0.12$ , while the displayed weather model uses $0.20$ .

Factor the Joint Distribution

The definition of conditional probability can be rearranged into the product rule:

p_{R,W}(r,w)=p_R(r)p_{W\mid R}(w\mid r).

The order can also be reversed:

p_{R,W}(r,w)=p_W(w)p_{R\mid W}(r\mid w).

For the rainy-and-windy cell, both routes give the same joint probability:

P(R=1)P(W=1\mid R=1) =0.30\left(\frac23\right) =0.20,

and:

P(W=1)P(R=1\mid W=1) =0.40(0.50) =0.20.

A joint distribution is therefore equivalent to a marginal distribution for one variable together with a conditional distribution for the other.

Exercise: Reconstruct a joint probability

Suppose $P(X=1)=0.25$ and $P(Y=1\mid X=1)=0.60$ . What is $P(X=1,Y=1)$ ?

Check the joint probability

Compute it first, then check your number.

HintUse a marginal and a conditional factor

Multiply $P(X=1)$ by $P(Y=1\mid X=1)$ .

SolutionApply the product rule

The joint probability is:

P(X=1,Y=1) =P(X=1)P(Y=1\mid X=1) =0.25(0.60) =0.15.

We first enter the group where $X=1$ , which has probability $0.25$ . Within that group, $60\%$ of the probability also has $Y=1$ , leaving joint probability $0.15$ .

More Variables Give More Possible Factorizations

For three variables, repeated use of the product rule gives:

p(x,y,z)=p(x)p(y\mid x)p(z\mid x,y).

For $n$ variables, the probability chain rule is:

p(x_1,\ldots,x_n) =p(x_1)\prod_{i=2}^{n}p(x_i\mid x_1,\ldots,x_{i-1}).

“The joint mass of the complete sequence equals the mass of the first value, multiplied by each later value’s probability given every value that came before it in the chosen order.” No independence assumption is being made.

This identity does not assume independence. It expresses one joint probability as a sequence of conditional probabilities. A different variable order gives a different-looking but equally valid factorization.

Autoregressive language models apply this identity to a token sequence:

p(t_1,\ldots,t_n) =p(t_1)\prod_{i=2}^{n}p(t_i\mid t_1,\ldots,t_{i-1}).

The model estimates the conditional factors; multiplying them gives the joint probability assigned to the complete sequence.

Exercise: Evaluate a three-variable chain factorization

Suppose $P(X=1)=0.5$ , $P(Y=1\mid X=1)=0.4$ , and $P(Z=1\mid X=1,Y=1)=0.25$ . What is $P(X=1,Y=1,Z=1)$ ?

Check the joint probability

Compute it first, then check your number.

HintMultiply in the stated order

Use $0.5(0.4)(0.25)$ .

SolutionMultiply the conditional stages

The chain rule gives:

P(X=1,Y=1,Z=1)=0.5(0.4)(0.25)=0.05.

Each factor conditions on the values that appear earlier in the chosen ordering.

Total Probability Recovers a Marginal

Combining the product rule with marginalization gives:

p_R(r)=\sum_w p_{R\mid W}(r\mid w)p_W(w).

For rain:

\begin{aligned} P(R=1) &=P(R=1\mid W=0)P(W=0)\\ &\quad+P(R=1\mid W=1)P(W=1)\\ &=\frac{1}{6}(0.60)+\frac{1}{2}(0.40)\\ &=0.10+0.20\\ &=0.30. \end{aligned}

The marginal is a weighted average of the conditional probabilities across all possible values of the removed variable.

Sums Become Integrals for Continuous Variables

If $X$ and $Y$ are continuous, a joint density $f_{X,Y}(x,y)$ assigns density over the plane. Probabilities come from integrating over regions rather than reading individual point values:

P((X,Y)\in D)=\iint_D f_{X,Y}(x,y)\,dx\,dy.

Marginal density is obtained by integrating out the other variable:

f_X(x)=\int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dy.

Where $f_Y(y)>0$ , conditional density is:

f_{X\mid Y}(x\mid y) =\frac{f_{X,Y}(x,y)}{f_Y(y)}.

The same three operations remain: keep variables jointly, integrate one out, or fix known information and renormalize. As with any continuous density, $f_{X,Y}(x,y)$ is not the probability of one exact point.

See What Each Operation Retains

The following experiment starts with a joint probability table. Its heatmap shows the paired structure, while the adjacent bars show the two marginals. Change the interior probabilities while keeping the total equal to one and observe how a joint distribution can change even when a marginal changes only slightly.

Inspect a joint table and its marginals

Edit the joint probabilities, keeping them nonnegative with total one. Compare the heatmap with the row and column sums.

Ready to run.

Exercise: Distinguish marginalization from conditioning

Which operation produces $p_X(x)=\sum_y p_{X,Y}(x,y)$ ?

Choose one

Marginalize over $Y$ by adding all of its possible values.

Condition on one fixed value of $Y$ and renormalize.

Select one choice, then check.

HintLook at the range of the sum

The formula includes every possible value of $Y$ , rather than selecting one.

SolutionSum out the variable that is no longer needed

This is marginalization. Adding $p_{X,Y}(x,y)$ over every $y$ preserves the total probability associated with $X=x$ while discarding which $Y$ value accompanied it.

Before Moving between Distribution Views

State which variables remain and which information is being removed or fixed. A joint distribution retains paired structure. A marginal sums or integrates a variable out and may lose dependence information. A conditional distribution selects a value or event and renormalizes the remaining variable. Check that every resulting distribution is nonnegative and normalized.

The next lesson studies independence. It identifies the special case in which the joint distribution factors into a product of marginals and conditioning on one variable does not change the distribution of the other.