Matrix Layer Backward Pass

The scalar rule dL/dw = g * x extends to a whole affine layer without changing its meaning.

Use the forward convention from earlier chapters:

X: (batch, inputs)
W: (inputs, outputs)
b: (outputs,)
Z = XW + b: (batch, outputs)

Let G = dL/dZ have the same shape as Z. The backward formulas are:

dL/dW = X^T G            shape (inputs, outputs)
dL/db = sum_rows(G)       shape (outputs,)
dL/dX = G W^T            shape (batch, inputs)

Each formula has an interpretation:

• X^T G adds input * arriving gradient over examples for every weight;
• sum_rows(G) adds the contribution to each shared output bias;
• G W^T sends the gradient toward each input feature.

Derive the entries before memorizing the matrices

Write one pre-activation entry as:

Z[n, o] = sum_i X[n, i] * W[i, o] + b[o]

Here n selects an example, i an input feature, and o an output feature. Let G[n, o] = dL/dZ[n, o].

One weight W[i, o] is reused by every example, so all batch contributions add:

dL/dW[i, o] = sum_n X[n, i] * G[n, o]

That indexed sum is exactly the (i, o) entry of X^T G. The broadcast bias b[o] is also reused by every example:

dL/db[o] = sum_n G[n, o]

Finally, one input X[n, i] contributes to every output o, so those output paths add:

dL/dX[n, i] = sum_o G[n, o] * W[i, o]

This is the (n, i) entry of G W^T. The transpose symbols are consequences of aligning these indices, not rules to guess from memory.

A Complete Two-Example Calculation

Let:

X = [[1, 2],
     [3, 1]]

W = [[ 2, -1],
     [ 1,  3]]

b = [1, -2]

The forward pre-activations are:

Z = XW + b
  = [[5,  3],
     [8, -2]]

Suppose the later graph sends:

G = dL/dZ
  = [[ 1, 2],
     [-1, 3]]

Then:

dL/dW = X^T G
      = [[1, 3],     [[ 1, 2],
         [2, 1]]  @   [-1, 3]]
      = [[-2, 11],
         [ 1,  7]]

dL/db = [1 + (-1), 2 + 3]
      = [0, 5]

dL/dX = G W^T
      = [[ 0, 7],
         [-5, 8]]

Shape checking catches many transpose errors:

X^T G: (inputs, batch) × (batch, outputs) -> (inputs, outputs)
G W^T: (batch, outputs) × (outputs, inputs) -> (batch, inputs)