PyTorch Computing Gradients
backward() is the call that computes gradients and stores them in .grad. One backward() call fills the .grad of every tracked input.
No gradient before backward()
The answer is ready, but no gradient exists yet. .grad still holds None.
import torch
x = torch.tensor(3.0, requires_grad=True)
y = (2 * x) ** 2
print(y.item())
print(x.grad)backward() fills .grad
One call computes the gradient and stores it. At x = 3 the gradient is 24.
y.backward()
print(x.grad.item())Check a second point by hand
By hand, the gradient of (2x) ** 2 is 8x. At x = 5, autograd and the hand rule agree on 40.
w = torch.tensor(5.0, requires_grad=True)
((2 * w) ** 2).backward()
print(w.grad.item())Example
import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (2 * x) ** 2
y.sum().backward()
print("Gradients:", x.grad)
print("Expected 8x:", 8 * torch.tensor([1.0, 2.0, 3.0]))Key points
- Scalar: a single number, not a row or a table
- Before backward(), .grad holds None
- Use backward() on a scalar answer to compute its gradients
- Use .sum() to turn a row of answers into one scalar for backward()
- The gradient of (2x) ** 2 is 8x, worked out by hand
- In short, backward() does the derivative work and .grad is where the answers land
In this example, x holds three points in one row. y applies (2x) ** 2 to each point. y is a row, so sum() first adds its values into one scalar. backward() then fills x.grad for all three points at once. The gradients print as 8, 16, and 24. The last line prints 8 times the three points, and the two rows match.
Try it Yourself
Edit the code, then press Run.
Running is simulated · Python Playground
Exercise
What does backward() store in .grad?