Layer Composition

An MLP composes layers.

For a one-hidden-layer MLP:

h = relu(xW1 + b1)
scores = hW2 + b2

Read this from left to right:

1. The input x enters the first affine map.
2. ReLU transforms the first layer's output into hidden activations h.
3. The second affine map turns h into output scores.

The whole model is still one function:

scores = f(x; W1, b1, W2, b2)

But internally it has stages.

Shape trace

Suppose:

x shape:  (3)
W1 shape: (3, 4)
b1 shape: (4)
W2 shape: (4, 2)
b2 shape: (2)

Then:

h shape:      (4)
scores shape: (2)

The hidden layer changes the representation from 3 input features to 4 hidden features. The output layer changes 4 hidden features to 2 scores.