Variance and Covariance
Variance measures spread.
If X has mean mu, then:
Large variance means values tend to be far from the mean. Small variance means values stay close.
Variance is always nonnegative because it averages squared distances from the mean.
Standard Deviation
Standard deviation is the square root of variance.
It uses the same units as the original variable.
If variance is 9, standard deviation is 3.
Covariance
Covariance measures how two variables move together.
If high values of X tend to appear with high values of Y, covariance is
positive.
If high values of X tend to appear with low values of Y, covariance is
negative.
If there is no linear relationship, covariance is near zero.
Covariance is not the same as causation. Two variables can move together because one causes the other, because both share another cause, or by accident in a small sample.
ML Reading
Variance appears in data spread, noise, uncertainty, initialization, and model evaluation.
Covariance appears when features move together.
Covariance matters because feature relationships affect models. If two features move together strongly, they may carry overlapping information.
Covariance also depends on scale. A feature measured in larger units can have a larger covariance simply because its numbers are larger. Correlation is often used when we want a normalized measure of linear co-movement.
Suppose X takes values 0 and 2 with equal probability.
The mean is 1. What is the variance?
Compute it first, then check your number.
Hint
Compute the average of (X - 1)^2.
Solution
For X = 0, (0 - 1)^2 = 1. For X = 2, (2 - 1)^2 = 1. The average is
1. Variance is the average squared distance from the mean.
If Var(X) = 9, what is the standard deviation?
Compute it first, then check your number.
Hint
Standard deviation is the square root of variance.
Solution
sqrt(9) = 3, so the standard deviation is 3. Standard deviation is the
square root of variance, returning to the original units.
If high values of X tend to appear with high values of Y, is covariance
positive or negative?
Answer it first, then check.
Hint
Variables moving together in the same direction have positive covariance.
Solution
The covariance is positive because the variables tend to move together. High values of one variable tend to appear with high values of the other.
Does positive covariance by itself prove that X causes Y?
Answer it first, then check.
Hint
Moving together is not the same as causation.
Solution
No. Positive covariance shows co-movement, not causation by itself. A shared cause or sampling accident could also create the pattern.
Enter 1 if covariance can change when the units or scale of a variable
change.
Compute it first, then check your number.
Hint
Ask what happens if every value of one variable is multiplied by 10.
Solution
Enter 1. Covariance uses the numerical scale of the variables, so rescaling a
variable changes the covariance value.
Before Moving On
Expectation gives center. Variance gives spread.