LLM Primer

Conditional Probability

Conditional probability measures probability after new information is known.

It is written:

P(A∣B)P(A \mid B)

Read this as: probability of A given B.

ABP(A | B): restrict attention to B
Conditional probability measures an event after restricting the sample space.

Formula

If P(B) > 0, then:

P(A∣B)=P(A∩B)P(B)P(A \mid B) = \frac{P(A \cap B)}{P(B)}

This means we restrict attention to cases where B happened, then ask how often A also happened.

The denominator changes from the whole sample space to the event B.

That is the main idea. Conditional probability is not a new kind of probability; it is probability after narrowing the context.

Small Table Example

Suppose 100 examples are known to be in group B. Among those 100 examples, 40 also have property A.

Then:

P(A∣B)=40/100=0.4P(A \mid B) = 40/100 = 0.4

ML Reading

Model predictions are often conditional probabilities.

For example:

P(label∣input)P(\text{label} \mid \text{input})

means the probability of a label given the input.

The condition matters. P(label) is a prior probability of a label before seeing the input. P(label | input) is the probability after seeing the input.

Language modeling uses:

P(next token∣previous tokens)P(\text{next token} \mid \text{previous tokens})

Changing the condition can change the probability. That is the point of conditioning. The same label, token, or event may be likely in one context and unlikely in another.

MATH-C07-T09-001Exercise: Compute a conditional probability

Suppose P(A and B) = 0.2 and P(B) = 0.5.

What is P(A | B)?

Compute it first, then check your number.

Hint

Divide P(A and B) by P(B).

Solution

P(A | B) = 0.2 / 0.5 = 0.4. We divide by P(B) because the condition tells us to look only inside the cases where B happened.

MATH-C07-T09-002Exercise: Restrict the denominator

Among 100 examples in group B, 40 also have property A.

What is P(A | B)?

Compute it first, then check your number.

Hint

Once we know B, count only examples inside B.

Solution

P(A | B) = 40 / 100 = 0.4. The denominator is the 100 examples already known to be in group B, not all possible examples.

MATH-C07-T09-003Exercise: Read the condition

In P(label | input), what is the condition?

Answer it first, then check.

Hint

The condition appears after the vertical bar.

Solution

The condition is input. We are asking for the label probability after seeing the input.

MATH-C07-T09-004Exercise: Compare prior and conditional

Is P(label | input) usually the same expression as P(label)?

Answer it first, then check.

Hint

One expression uses the input as context; the other does not.

Solution

No. P(label) is not conditioned on the input, while P(label | input) is. The input can change the context, and therefore the probability.

MATH-C07-T09-005Exercise: Condition changes context

Enter 1 if changing the condition in P(label | input) can change the label probability.

Compute it first, then check your number.

Hint

Compare a label before and after seeing an input.

Solution

Enter 1. The condition is information already known. Changing that information can change the probability.

Before Moving On

Conditional probability is probability with context.