Gradients With Respect to Biases

For:

z = wx + b

the local derivative with respect to the bias is:

dz/db = 1

So if:

g = dL/dz

then:

dL/db = g

The bias gradient is the upstream gradient itself.

For a batch, bias gradients from examples add. If three examples contribute:

[2, -1, 4]

then the batch bias gradient is:

2 + (-1) + 4 = 5

Exercise: Single bias gradient

If dL/dz = -3, what is dL/db for z = wx + b?

Your answer

Compute it first, then check your number.

HintBias adds directly

The derivative of z with respect to b is 1.

SolutionWork it out

dL/db = dL/dz * dz/db = -3 x 1 = -3.

Exercise: Batch bias accumulation

Bias-gradient contributions are [2, -1, 4]. What is the accumulated bias gradient?

Your answer

Compute it first, then check your number.

HintAdd contributions

Batch contributions to the same bias add.

SolutionWork it out

2 + (-1) + 4 = 5.