LLM Primer

Summary and Revision Notes

Use this page to review the chapter before doing the exercises.

Key Ideas

IdeaMeaningQuestion to ask
sample spaceall possible outcomesWhat can happen?
eventa set of outcomesWhich outcomes are included?
probabilitynumber from 0 to 1 assigned to an eventIs this value consistent?
random variablefunction from outcomes to valuesWhat number does each outcome produce?
distributionprobability assignment over possible valuesDo probabilities sum to 1?
expectationprobability-weighted averageWhat is the long-run average?
varianceexpected squared distance from the meanHow spread out are values?
covariancejoint movement between variablesDo the variables move together?
independenceone event does not change another probabilityDoes knowing one event change the other?
conditional probabilityprobability after context is knownWhat is the new denominator?
Bayes' ruleupdate after evidenceHow do prior, likelihood, and evidence combine?

Formulas to Remember

Complement:

P(not A)=1−P(A)P(\text{not } A) = 1 - P(A)

Addition with overlap:

P(A∪B)=P(A)+P(B)−P(A∩B)P(A \cup B) = P(A) + P(B) - P(A \cap B)

Expectation:

E[X]=∑xxP(X=x)E[X] = \sum_x xP(X=x)

Variance:

Var(X)=E[(X−μ)2]Var(X) = E[(X - \mu)^2]

Independence:

P(A∩B)=P(A)P(B)P(A \cap B) = P(A)P(B)

Conditional probability:

P(A∣B)=P(A∩B)P(B)P(A \mid B) = \frac{P(A \cap B)}{P(B)}

Bayes' rule:

P(A∣B)=P(B∣A)P(A)P(B)P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}

Useful Distinctions

Do not confuseDifference
event and random variablean event is a set; a random variable maps outcomes to values
probability and expectationprobability measures event likelihood; expectation is an average value
independent and mutually exclusiveindependent events can both happen; mutually exclusive events cannot
prior and posteriorprior is before evidence; posterior is after evidence
covariance and causationcovariance is co-movement, not proof of cause
Bernoulli and categoricalBernoulli is binary; categorical chooses one class from several
counting and adding probabilitycount outcomes only when they are equally likely
distribution name and assumptiona named distribution encodes a modeling choice

Common Traps

  • Treating probability as certainty.
  • Forgetting probabilities must stay between 0 and 1.
  • Forgetting probabilities in a discrete distribution must sum to 1.
  • Counting outcomes when assigned probabilities are unequal.
  • Treating an expected value as the next guaranteed observation.
  • Confusing independent events with mutually exclusive events.
  • Ignoring the conditioning information in P(A | B).
  • Forgetting that Bayes' rule normalizes by the probability of evidence.
  • Treating distribution names as labels instead of modeling assumptions.
  • Treating a mini-batch average as exact expected loss.
  • Reading covariance without considering scale.

Mental Model

Probability turns uncertainty into quantities we can reason about, compute with, compare, and update after evidence.