Parameters

Parameters are the trainable tensors of a model.

A parameter is not merely a tensor with numbers. It is a tensor that the model expects the optimizer to update.

from torch import nn

layer = nn.Linear(3, 2)

for name, parameter in layer.named_parameters():
    print(name, parameter.shape)

For a linear layer with 3 input features and 2 output features, the weight has shape (2, 3) and the bias has shape (2).

That means:

weight entries = 2 * 3 = 6
bias entries   = 2
total          = 8

Counting parameters is a simple way to keep models concrete. A large model may have billions of parameters, but every parameter is still an entry in a tensor that participates in forward computation and receives updates during training.

PyTorch modules can list their parameters because the parameters are registered inside the module. That registration is why an optimizer can be given model.parameters() instead of a manually written list of every weight.

Parameter counting is not only bookkeeping. It helps you notice when a model is larger than the data or hardware can support.