Singular Value Decomposition
Singular value decomposition, or SVD, rewrites a matrix as:
This says the matrix can be read as:
- change directions with
V^T - stretch or shrink by
Sigma - change directions with
U
SVD is like a more general cousin of diagonalization. Diagonalization looks for eigenvector coordinates, but it works cleanly only for some square matrices. SVD works for any real matrix, including rectangular matrices.
Why SVD Is Useful
SVD works for any real matrix, even when the matrix is not square.
It reveals:
- important input directions
- important output directions
- strengths of those directions
- which directions are weak or nearly unused
The singular values in Sigma are nonnegative and usually ordered from largest
to smallest.
If a singular value is large, the matrix strongly carries that direction. If a singular value is zero, that direction is collapsed.
A Useful Reading
Think of SVD as three questions:
- which input directions matter?
- how strongly does the matrix carry each direction?
- which output directions do those inputs become?
The singular values answer the middle question. They are strengths.
This is different from asking only for eigenvectors. SVD separates input directions from output directions, so it still works when the matrix maps one kind of space into another, or when the matrix is rectangular.
ML Reading
Large singular values often point to directions where the matrix has strong effect.
Small singular values often point to directions that are weak, noisy, or nearly collapsed.
This is why SVD appears in compression, dimensionality reduction, numerical stability, and model inspection.
In A = U Sigma V^T, which part contains the singular values?
Enter U, Sigma, or V^T.
Answer it first, then check.
Hint
The middle factor stretches or shrinks directions.
Solution
Sigma contains the singular values. They measure the strength of each
direction, while V^T and U describe the input and output directions.
If the singular values are 10, 2, and 0.1, which direction is strongest?
Compute it first, then check your number.
Hint
The largest singular value marks the strongest direction.
Solution
The direction with singular value 10 is strongest because it is scaled the
most.
What does a singular value of 0 mean?
Enter collapsed or strong.
Answer it first, then check.
Hint
A zero strength means no output remains in that direction.
Solution
A singular value of 0 means that direction is collapsed. The matrix carries no
nonzero output strength along that singular direction.
Enter 1 if SVD can be used for rectangular real matrices, not only square
matrices.
Compute it first, then check your number.
Hint
SVD separates input directions, strengths, and output directions.
Solution
Enter 1. SVD works for rectangular real matrices because it can describe
input directions, output directions, and strengths separately.
Before Moving On
SVD turns a matrix into directions and strengths.