Capacity as a Function Family
A model architecture defines a family of functions.
The parameters choose one function from that family.
For:
f(x; w, b) = wx + b
each pair (w, b) gives a different line.
w = 1, b = 0 f(x) = x
w = 2, b = 0 f(x) = 2x
w = 2, b = 3 f(x) = 2x + 3
This family can represent many lines, but not every possible pattern. It cannot bend. It cannot make a curve. It cannot make several separated regions by itself.
That limitation is not a failure. It is a clue.
Later chapters add nonlinear activations and layers so the model family becomes richer.
Capacity is not always better
A richer family can fit more patterns. It can also fit noise when the data is limited.
For now, keep the simple definition:
capacity = what kinds of functions the model family can represent
For f(x; w, b) = wx + b, with w = 2 and b = 3, what is f(4)?
Compute it first, then check your number.
HintParameters choose the line
Use the current w and b.
SolutionWork it out
f(4) = 2 x 4 + 3 = 11.
A model family f(x) = wx + b can represent straight lines. Enter 1 if it can represent every curved pattern without changing the architecture, or 0 if it cannot.
Compute it first, then check your number.
HintThink about shape
Changing w and b changes the line, but it is still a line.
SolutionWork it out
The family wx + b contains lines. Changing parameters changes which line,
but it does not turn the architecture into a curved model family.