Depth and Width
Two simple words describe the size of an MLP.
Depth is the number of layers with learned parameters.
Width is the number of units in a layer.
For example:
input size: 3
hidden layer 1: 4 units
hidden layer 2: 5 units
output layer: 2 units
This network has three learned layers: two hidden layers and one output layer. Its hidden widths are 4 and 5.
The input size describes the data. It is not usually counted as a learned layer.
Why width matters
A wider hidden layer can compute more hidden activations at the same stage. Those activations can act like a collection of detectors or features.
Why depth matters
Depth lets the model compose transformations. Earlier layers build simpler representations. Later layers combine them.
An MLP has input size 6, hidden layers with 8 and 4 units, and an output layer with 3 units. How many learned layers does it have?
Compute it first, then check your number.
HintDo not count the input size
Count layers with parameters.
SolutionWork it out
The network has hidden layer 1, hidden layer 2, and the output layer. That
is 3 learned layers.
A hidden layer has weight matrix shape (6, 10), using (input_features, hidden_units) order. What is the width of the hidden layer?
Compute it first, then check your number.
HintLook at hidden_units
The shape order is given in the question.
SolutionWork it out
The weight shape is (6, 10), where the second dimension is hidden units.
The hidden width is 10.