LLM Primer

Matrices in Neural Network Layers

A basic neural network layer uses a matrix.

One common form is:

h=σ(Wx+b)h = \sigma(Wx + b)

Read it as:

multiply input x by weight matrix W
add bias b
apply activation function sigma
xWWx+ bhweightsactivationh = activation(Wx + b)
A neural layer is built from matrix-vector multiplication plus a few extra steps.

Shapes

If:

x has length 3
W has shape 2 x 3
b has length 2

then:

Wx has length 2
Wx + b has length 2
h has length 2

The weight matrix controls how input features are mixed into output features.

Read each output coordinate as one learned feature detector. One row of WW decides how to mix the input features for that output coordinate. The bias then shifts that coordinate before the activation function changes it.

MATH-C03-T09-001Exercise: Layer output length

If xx has length 4 and WW has shape 6 x 4, what is the length of WxWx?

Compute it first, then check your number.

HintUse the rows

The output length is the number of rows of WW.

SolutionOutput length

The input length 4 matches the number of columns of WW. The output length is the number of rows: 6.

Rows Become Output Features

Each row of WW computes one weighted sum of the input.

For example, if:

W=[120−103]W = \begin{bmatrix} 1 & 2 & 0 \\ -1 & 0 & 3 \end{bmatrix}

and:

x=[456]x = \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix}

then the first entry of WxWx is:

1⋅4+2⋅5+0⋅6=141\cdot4 + 2\cdot5 + 0\cdot6 = 14

The second entry is:

−1⋅4+0⋅5+3⋅6=14-1\cdot4 + 0\cdot5 + 3\cdot6 = 14

So:

Wx=[1414]Wx = \begin{bmatrix} 14 \\ 14 \end{bmatrix}
MATH-C03-T09-002Exercise: Compute one pre-activation

For the WW and xx above, what is the first entry of WxWx?

Compute it first, then check your number.

HintUse the first row

Compute 1⋅4+2⋅5+0⋅61\cdot4 + 2\cdot5 + 0\cdot6.

SolutionFirst weighted sum
1⋅4+2⋅5+0⋅6=4+10+0=141\cdot4 + 2\cdot5 + 0\cdot6 = 4 + 10 + 0 = 14

This is one pre-activation value. The first row of WW chooses how to mix the three input features for the first output coordinate.

Bias And Activation

The matrix product WxWx gives pre-activation values.

The bias bb shifts those values. The activation function σ\sigma then changes them coordinate by coordinate.

If:

Wx=[1414]b=[1−2]Wx = \begin{bmatrix} 14 \\ 14 \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

then:

Wx+b=[1512]Wx+b = \begin{bmatrix} 15 \\ 12 \end{bmatrix}
MATH-C03-T09-003Exercise: Add a bias vector

If Wx=[14,14]TWx = [14,14]^T and b=[1,−2]Tb = [1,-2]^T, what is the second entry of Wx+bWx+b?

Compute it first, then check your number.

HintAdd matching entries

Use the second coordinate of both vectors.

SolutionBias addition
[14,14]T+[1,−2]T=[15,12]T[14,14]^T + [1,-2]^T = [15,12]^T

The second entry is 12. Bias addition is coordinate-wise: the second bias shifts the second pre-activation from 14 down by 2.

Why The Matrix Matters

This pattern is small, but it appears throughout deep learning.

Transformers use many matrix multiplications: projections, feed-forward layers, attention scores, and output logits all rely on the same shape discipline.

The names change, but the reading habit remains:

check the shape
read rows as weighted sums
track the output length
MATH-C03-T09-004Exercise: Spot the layer mismatch

Enter 1 if a length-5 input can multiply a weight matrix with shape 3 x 4, or 0 if it cannot.

Compute it first, then check your number.

HintCheck columns

A 3 x 4 matrix expects a length-4 input vector.

SolutionInput mismatch

Enter 0. A 3 x 4 matrix has 4 columns, so it expects a length-4 input. A length-5 input does not fit.

Next, we close the chapter and collect the important rules.