PyTorch What Is Autograd

Autograd is the part of PyTorch that tracks operations on tensors. A gradient is how much the answer moves when an input moves a little.

Track one squaring

x is created with tracking on. The print shows the result and the recorded operation name.

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
print(y)

Ask for the gradient

backward() works back through the record. It stores the gradient in x.grad.

y.backward()
print(x.grad)

Read the gradient as a number

.item() pulls the gradient out as a plain Python float. The answer moves 6 for each 1 that x moves, at x = 3.

print(x.grad.item())

Example

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
print("y:", y)
y.backward()
print("Gradient:", x.grad)
print("As a number:", x.grad.item())

Key points

Gradient: how much the answer moves when an input moves a little
requires_grad=True: the switch that tells autograd to track a tensor
Autograd records each operation as it runs
Use backward() on the answer to fill in the gradients
.grad holds the gradient for a tracked tensor after backward()
For y = x ** 2, the gradient is 2 * x
In short, autograd watches the maths and reports how each input moves the answer

In this example, x is a tensor holding 3.0 with tracking on. y squares x, so y holds 9.0. Autograd records the squaring as it runs. y.backward() works back through that record. It fills x.grad with the gradient at x = 3. The gradient prints as 6, and .item() reads it as the plain number 6.0.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
print("y:", y)
y.backward()
print("Gradient:", x.grad)
print("As a number:", x.grad.item())

Exercise

For y = x ** 2 at x = 3, what gradient does autograd report?

3 9 6 2