Broadcasting
Broadcasting lets a smaller array act across a larger array when the meaning is clear.
The most common early example is adding a bias vector to every row of a batch.
X shape: (3, 2)
b shape: (2)
X + b shape: (3, 2)
If:
X = [
[10, 20],
[30, 40],
[50, 60]
]
b = [1, -1]
then:
X + b = [
[11, 19],
[31, 39],
[51, 59]
]
The vector b is applied to each row.
Broadcasting is not magic
Broadcasting is a rule for reusing values along compatible axes. It is helpful because it avoids writing repeated copies by hand.
But the rule must match the meaning. A bias with shape (2) can add to rows with 2 features. A bias with shape (3) cannot add to rows with 2 features.
(3, 2) + (2) valid
(3, 2) + (3) not valid in the intended feature-wise sense
A batch has shape (5, 4). You want one bias value per feature. What shape should the bias vector have?
Compute it first, then check your number.
HintFeature axis
The feature dimension is the second number in (5, 4).
SolutionWork it out
The batch has 5 examples and 4 features per example. One bias per feature
means the bias vector has shape (4).
X has shape (7, 3) and b has shape (3). What is the shape of X + b?
Compute it first, then check your number.
HintThe batch stays
Adding the bias does not remove examples from the batch.
SolutionWork it out
b supplies one value for each of the 3 features. It is added to every one
of the 7 rows, so the output shape remains (7, 3).