PCA
Principal component analysis, or PCA, finds directions where centered data varies most.
The first principal direction captures the strongest variation. The next principal direction captures the strongest remaining variation, subject to being orthogonal to the first.
Center First
PCA usually begins by centering the data: subtract the mean from each feature.
This moves the coordinate system so variation is measured around the center of the dataset.
Without centering, PCA can confuse the location of the data cloud with the directions in which the data varies.
PCA as Compression
If most variation lies along a few directions, PCA can project data onto those directions.
That gives a lower-dimensional representation.
The representation may be easier to visualize, store, or feed into another method. But PCA is linear, so it can miss nonlinear structure.
For example, if points lie near a straight diagonal band, one principal component may summarize much of the data. If points lie on a curved shape, one straight direction may not describe the structure well.
ML Reading
PCA is useful for understanding the idea of representation compression before we meet deeper nonlinear models.
It also reinforces a recurring question:
Which directions matter most?
PCA is also a good place to practice restraint. The first principal component is the direction of largest variance, not necessarily the most meaningful causal factor, the most fair feature, or the best feature for prediction.
Variance is a geometric fact about the data cloud. Meaning is a modeling and domain question. PCA can suggest structure, but the interpretation still has to be checked.
If most centered data points stretch along a diagonal line, which direction will the first principal component follow?
Enter 1 for the strongest variation direction, 0 for a random direction.
Compute it first, then check your number.
Hint
PCA follows variance, not a random axis.
Solution
The first principal component follows the strongest variation direction. Enter
1. PCA chooses a direction from the geometry of centered data, not a random
axis chosen in advance.
What common preprocessing step subtracts the mean from each feature before PCA?
Answer it first, then check.
Hint
It moves the data cloud so variation is measured around its center.
Solution
The step is centering. We subtract the mean from each feature so PCA measures variation around the data cloud's center instead of confusing location with spread.
If two principal components keep most of the variation from ten original features, how many coordinates does the compressed representation use?
Compute it first, then check your number.
Hint
The compressed representation keeps one coordinate per retained component.
Solution
It uses 2 coordinates, one for each retained principal component. The original
ten features are being represented through two chosen variation directions.
Does the first principal component always represent the most causal factor?
Answer it first, then check.
Hint
PCA follows variance, not causality.
Solution
No. PCA finds directions of large variance. That direction may or may not have a causal interpretation.
Enter 1 if a high-variance PCA direction can be important geometrically
without being the most meaningful or causal factor.
Compute it first, then check your number.
Hint
Separate geometric variance from semantic explanation.
Solution
Enter 1. PCA identifies directions of large variance in centered data. Those
directions may be useful, but their meaning is not guaranteed by PCA alone.
Before Moving On
PCA is a linear method for finding important directions in centered data.