Representation Before Prediction
An MLP does not usually jump from raw input directly to final prediction.
It first builds a hidden representation.
For a one-hidden-layer MLP:
h = relu(xW1 + b1)
scores = hW2 + b2
The hidden vector h is a learned representation of the input. It is not the original data. It is the data after the first layer has transformed it.
This is one reason neural networks are useful. They can learn intermediate features that make the final prediction easier.
Small interpretation
If a hidden unit outputs a large positive value after ReLU, it means its weighted pattern was present strongly enough for that input.
The output layer can combine several such hidden responses.
score = 2h_1 - h_2 + 0.5h_3 + b
The final score is built from learned hidden features, not only raw input features.
A model maps input shape (4) to hidden shape (6), then to output shape (2). What is the hidden representation size?
Compute it first, then check your number.
HintMiddle shape
The hidden representation is between input and output.
SolutionWork it out
The middle shape is (6), so the hidden representation size is 6.
Let h = [3, 1], output weights [2, -4], and bias 1. Compute the score.
Compute it first, then check your number.
HintUse hidden values as features
Compute h . w + b.
SolutionWork it out
3 x 2 + 1 x (-4) + 1 = 6 - 4 + 1 = 3.