Linear Independence and Basis
Vectors are linearly independent when none of them can be built from the others.
If one vector is already reachable from the rest, it is redundant.
For example:
So [1, 0] and [2, 0] do not give two independent directions. They both point
along the same line.
This is not about whether the numbers look different. It is about whether a vector adds a direction that was not already available.
For example:
are independent in the plane. Neither vector can be stretched or flipped to make the other. Together they give horizontal and vertical movement.
But:
are not independent as a set, even though they can still reach the whole plane. The second vector is extra. It repeats the horizontal direction.
Basis
A basis is a set of independent vectors that spans a space.
That means a basis has two jobs:
- it reaches the whole space
- it avoids unnecessary directions
In two dimensions, [1, 0] and [0, 1] form a basis for the plane.
The word "basis" matters because it gives every vector in the space a unique coordinate recipe. If a set is independent and spans the space, each target has one set of weights.
If a set spans but is not independent, there are multiple recipes for some targets. That is useful sometimes, but it is no longer a clean coordinate frame.
This is the key practical difference. A spanning set answers, "Can I reach the target?" A basis also answers, "Is the recipe unique?" The uniqueness is what makes basis coordinates dependable.
Why This Matters
Machine learning often deals with many features. Some features may repeat information already present in other features.
Independence helps us ask whether a direction adds new information or only repackages an old direction.
This question appears when we inspect feature columns, embeddings, activations, and learned representations. A model may have many coordinates, but some directions may still be redundant.
Can [2, 0] be made from [1, 0] by scalar multiplication?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
Ask whether one number times [1, 0] gives [2, 0].
Solution
Yes. 2[1, 0] = [2, 0], so enter 1. The second vector does not add a new
direction; it repeats the same horizontal direction with a different scale.
Can [1, 0] be made from [0, 1] by scalar multiplication?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
Scaling [0, 1] keeps the first coordinate equal to 0.
Solution
No. Any scalar multiple of [0, 1] has the form [0, c], so it cannot equal
[1, 0].
The vectors [1, 0], [2, 0], and [0, 1] span the plane.
Do they form a basis for the plane?
Answer it first, then check.
Hint
A basis must span and avoid redundancy.
Solution
No. The set spans the plane, but [2, 0] = 2[1, 0], so the set is redundant and
not a basis.
In the set [1, 0], [0, 1], [1, 1], which vector can be made from the first
two?
Enter [1, 1], [1, 0], or [0, 1].
Answer it first, then check.
Hint
Add the first two vectors.
Solution
[1, 1] = [1, 0] + [0, 1], so [1, 1] is reachable from the first two vectors.
It is therefore redundant in this set, even though it has useful-looking
coordinates.
Enter 1 if a basis gives each vector in the space one coordinate recipe.
Compute it first, then check your number.
Hint
Spanning reaches the vector. Independence removes duplicate recipes.
Solution
Enter 1. A basis spans the space and avoids redundancy, so each vector has one
coordinate recipe in that basis.
Before Moving On
A basis is a non-redundant coordinate frame for a space.