Gradients With Respect to Weights

Weights control how much an input contributes to a pre-activation.

For one scalar unit:

z = wx + b

the local derivative with respect to the weight is:

dz/dw = x

If the upstream gradient is:

g = dL/dz

then:

dL/dw = g x
z = wx + bupstream g = dL/dzxwbdL/dx = g wdL/dw = g xdL/db = g
For z = wx + b, the same upstream gradient produces gradients for x, w, and b.

The weight gradient says how changing that weight would change the loss, using the current input and upstream signal.

Exercise: Weight gradient

For z = wx + b, let x = 4 and upstream gradient g = dL/dz = 3. What is dL/dw?

Compute it first, then check your number.

HintUse g x

The local derivative dz/dw is x.

SolutionWork it out

dL/dw = g x = 3 x 4 = 12.

Exercise: Input affects weight gradient

If g = 5 and x = 0, what is dL/dw?

Compute it first, then check your number.

HintMultiply by input

The weight gradient is proportional to the input value.

SolutionWork it out

dL/dw = g x = 5 x 0 = 0.