Hidden Layers

A hidden layer is a layer between the input and the output.

It is called hidden because its values are not the final prediction. They are intermediate activations used by later layers.

For one hidden layer:

h = relu(xW1 + b1)
scores = hW2 + b2

The first line builds hidden activations. The second line uses those activations to compute output scores.

A tiny example

Let:

x = [2, 1]
W1 = [
  [1,  0],
  [2, -1]
]
b1 = [0, 1]

First compute:

xW1 + b1 = [2 x 1 + 1 x 2, 2 x 0 + 1 x (-1)] + [0, 1]
          = [4, -1] + [0, 1]
          = [4, 0]

Then apply ReLU:

h = [4, 0]

The hidden layer produced two hidden activations.