Parameters, Activations, and Outputs as Tensors
In deep learning, many important objects are tensors.
The input data is a tensor. The learned parameters are tensors. The intermediate values inside the model are tensors. The final output is a tensor.
The names help separate their roles.
Parameters
Parameters are values the model learns.
For a simple layer:
XW + b
the weight matrix W and bias vector b are parameters.
If:
X shape: (batch, input_features)
W shape: (input_features, output_features)
b shape: (output_features)
then:
XW + b shape: (batch, output_features)
Activations
Activations are intermediate values produced as data moves through the model.
The word can mean two closely related things:
- the output values of a layer;
- the values after applying an activation function.
Context usually makes the meaning clear.
Outputs
The output tensor is what the model gives back for the task at hand.
For a classifier, the output may contain one score per class. For a language model, the output may contain one score per vocabulary item at each position.
The same habit applies: read the shape and name the axes.
A layer maps 6 input features to 4 output features. What is the shape of W in (input_features, output_features) order?
Compute it first, then check your number.
HintUse the convention
The question gives the order: (input_features, output_features).
SolutionWork it out
There are 6 input features and 4 output features, so W has shape (6, 4).
The same layer produces 4 output features. What is the size of the bias vector b?
Compute it first, then check your number.
HintBias follows output
The bias is added after the layer computes its output features.
SolutionWork it out
The layer produces 4 output features, so b has shape (4).
X has shape (8, 6), W has shape (6, 4), and b has shape (4). What is the shape of XW + b?
Compute it first, then check your number.
HintMultiply first
(8, 6) x (6, 4) gives (8, 4). Then the bias broadcasts across rows.
SolutionWork it out
The matrix product XW has shape (8, 4). The bias vector has one value
per output feature, so it broadcasts across the 8 examples. The activation
shape is (8, 4).