LLM Primer

Rank and Dimension

Rank measures how many independent output directions a matrix can produce.

If a matrix maps many input directions into the same output direction, the rank is low.

input spaceuseful directionnull directionAcolumn spacerank = visible outputs
Rank is what remains visible after a matrix maps inputs to outputs.

Rank as Visible Output Dimension

Consider:

A=[1200]A = \begin{bmatrix} 1 & 2 \\ 0 & 0 \end{bmatrix}

The second row is zero. Every output has the form [number, 0].

So the outputs lie on a line, not the whole plane. The rank is 1.

This matrix accepts two input coordinates, but only one output direction survives. The input may change in two ways. The output can only move left or right.

Rank is therefore not just the number of rows or columns. It is the number of independent directions actually used by the outputs.

Rank From Columns

Rank can be read from the independent columns of a matrix.

For:

B=[1200]B = \begin{bmatrix} 1 & 2 \\ 0 & 0 \end{bmatrix}

the columns are [1, 0] and [2, 0]. The second column is twice the first, so there is only one independent output direction. The rank is 1.

For:

I=[1001]I = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}

the columns are independent. They reach the whole plane, so the rank is 2.

Why Rank Matters

Rank tells how much independent structure survives the matrix.

Low rank often means redundancy or compression. High rank means more independent directions are preserved.

In ML, this shows up when we compress activations, inspect embeddings, factorize large matrices, or ask whether a representation uses many directions or only a few.

Rank is not automatically good or bad. A low-rank matrix may be useful if it removes noise or compresses repeated structure. It may be harmful if the lost directions contain rare but important information.

MATH-C05-T05-001Exercise: Read a rank

The matrix

[1000]\begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}

maps every input [x, y] to [x, 0].

What is its rank?

Compute it first, then check your number.

Hint

Its outputs lie on one line.

Solution

The output has only one independent direction: horizontal movement of the first coordinate. The second coordinate never survives in the output, so the rank is 1.

MATH-C05-T05-002Exercise: Read rank from repeated columns

The matrix has columns [1, 0] and [3, 0].

What is the rank?

Compute it first, then check your number.

Hint

Ask how many independent directions those columns provide.

Solution

Both columns lie on the same horizontal line, and [3, 0] = 3[1, 0]. There is one independent direction, so the rank is 1.

MATH-C05-T05-003Exercise: Read full rank in the plane

The identity matrix has columns [1, 0] and [0, 1].

What is its rank?

Compute it first, then check your number.

Hint

The two columns give two independent directions in the plane.

Solution

The columns are independent and span the whole plane, so the rank is 2.

MATH-C05-T05-004Exercise: Rank is not column count

A 2 x 3 matrix has columns [1, 0], [2, 0], and [3, 0].

What is its rank?

Compute it first, then check your number.

Hint

Count independent directions, not columns.

Solution

All three columns are scalar multiples of [1, 0], so they provide only one independent direction. The rank is 1.

MATH-C05-T05-005Exercise: Rank is task-dependent

Enter 1 if low rank can be helpful for compression but harmful when discarded directions matter for the task.

Compute it first, then check your number.

Hint

Ask what information lives in the directions that are removed.

Solution

Enter 1. Low rank is useful when the discarded directions are unimportant for the task. It can be harmful when those directions carry rare or useful detail.

Before Moving On

Rank is the dimension of the output space a matrix actually uses.