Hidden Units as Detectors

A hidden unit can be read as a small detector.

It computes a weighted sum:

z = x . w + b

Then it applies an activation:

a = relu(z)

If the weighted evidence is strong enough, the unit turns on. If not, it may output zero.

This is only an analogy, but it is useful. A hidden unit can respond to a pattern in the input. Later layers combine those responses into more complex predictions.

Small example

Let:

x = [2, 1]
w = [3, -1]
b = -4

Then:

z = 2 x 3 + 1 x (-1) - 4
  = 1

and:

relu(z) = 1

The unit is active.