PyTorch Model Parameters

Model parameters are the numbers a model holds for training to adjust. PyTorch lists them with parameters() and named_parameters().

List the parameters

One linear layer holds two parameters. The weight comes first, then the bias.

torch.manual_seed(0)
layer = nn.Linear(4, 3)
for param in layer.parameters():
    print(param.shape)

Name the parameters

Named parameters pair each name with its tensor. numel() counts the numbers in one tensor.

for name, param in layer.named_parameters():
    print(name, param.numel())

The saved form

state_dict is a dictionary that maps each name to its tensor. It is the form used when a model is saved.

print(list(layer.state_dict().keys()))
print(layer.state_dict()["weight"].shape)

Example

import torch
from torch import nn

torch.manual_seed(0)

model = nn.Sequential(
    nn.Linear(4, 3),
    nn.ReLU(),
    nn.Linear(3, 2),
)

total = 0
for name, param in model.named_parameters():
    count = param.numel()
    total += count
    print(f"{name}: shape {tuple(param.shape)}, count {count}")
print("Total parameters:", total)

Key points

Parameter: a tensor in a model that training can adjust
Use model.parameters() to list every parameter tensor
Use model.named_parameters() to list each name with its tensor
Use param.numel() to count the numbers in one parameter
Counting rule: a Linear(4, 3) layer holds 12 weights and 3 biases, so 15 in all
state_dict: a dictionary of the model's parameter names and tensors

In this example, the seed is set first, so the starting weights are repeatable. The model stacks two linear layers with a ReLU between them. The loop walks named_parameters() in order. Each line prints one name, its shape, and its count from numel(). The loop adds every count into a running total. The first layer holds 12 weights and 3 biases. The last layer holds 6 weights and 2 biases. ReLU holds no parameters, so it adds nothing. The total prints as 23. The state_dict of this model holds these same tensors under these same names. Saving that form to a file is taught later in the course.

Note

ReLU holds no parameters, so it adds nothing to a count

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

import torch
from torch import nn

torch.manual_seed(0)

model = nn.Sequential(
    nn.Linear(4, 3),
    nn.ReLU(),
    nn.Linear(3, 2),
)

total = 0
for name, param in model.named_parameters():
    count = param.numel()
    total += count
    print(f"{name}: shape {tuple(param.shape)}, count {count}")
print("Total parameters:", total)

Exercise

How many parameters does nn.Linear(4, 3) hold in total?

4 7 12 15