🎓 A1Learner
🔍
0 of 79 lessons completed

PyTorch Other Regression Losses

L1 loss is a loss that averages the absolute differences between predictions and targets. Smooth L1 loss sits between L1 loss and MSE.

L1 by hand

The absolute differences are 0, 0, and 10. Their mean is the L1 loss.

import torch

predictions = torch.tensor([3.0, 3.0, 13.0])
targets = torch.tensor([3.0, 3.0, 3.0])
print((predictions - targets).abs().mean())

L1 from the library

nn.L1Loss repeats that same average in one call.

import torch

predictions = torch.tensor([3.0, 3.0, 13.0])
targets = torch.tensor([3.0, 3.0, 3.0])
loss_fn = torch.nn.L1Loss()
print(loss_fn(predictions, targets))

Smooth L1 on the same miss

Smooth L1 scores the big miss a little lower than L1 does here.

import torch

predictions = torch.tensor([3.0, 3.0, 13.0])
targets = torch.tensor([3.0, 3.0, 3.0])
loss_fn = torch.nn.SmoothL1Loss()
print(loss_fn(predictions, targets))

Example

import torch

predictions = torch.tensor([3.0, 3.0, 3.0, 13.0])
targets = torch.tensor([3.0, 3.0, 3.0, 3.0])

mse = torch.nn.MSELoss()(predictions, targets)
l1 = torch.nn.L1Loss()(predictions, targets)
smooth = torch.nn.SmoothL1Loss()(predictions, targets)

print(f"MSE loss: {mse.item():.4f}")
print(f"L1 loss: {l1.item():.4f}")
print(f"Smooth L1 loss: {smooth.item():.4f}")
print("The outlier miss is 10.")

Key points

  • Regression: predicting a plain number, like a price or a score
  • Absolute difference: the distance between prediction and target, always positive
  • Use nn.L1Loss() for a loss that scores every miss by its distance alone
  • One big miss grows MSE fast, because MSE squares it
  • Use nn.SmoothL1Loss() for small misses scored like MSE and big misses scored like L1
  • All three losses score the same misses, and they disagree on how hard a big miss counts

In this example, three predictions hit their targets exactly. The fourth prediction misses by 10, and that miss is an outlier. An outlier is a value far away from the rest. MSE squares the miss, so one miss of 10 becomes 100 before averaging. L1 keeps the miss at 10, so its average stays much lower. Smooth L1 lands just below L1 on this data.

Note

The cutoff in SmoothL1Loss is called beta, and its default value is 1

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Which loss grows fastest when one prediction misses by a very large amount?