🎓 A1Learner
🔍
0 of 71 lessons completed

PyTorch Benchmarking Honestly

Benchmarking is timing code in a way that gives a number worth trusting. An honest benchmark warms up first, repeats the work, and reports the middle value.

One cold call, timed once

The first call in a fresh program pays startup costs. Its time mixes those costs with the real work.

import statistics
import time

import torch

torch.manual_seed(0)
a = torch.randn(512, 512)
b = torch.randn(512, 512)

start = time.perf_counter()
c = a @ b
ms = (time.perf_counter() - start) * 1000
print(f"One cold call: {ms:.0f} ms")

Warm up, repeat, take the median

Three warmup calls run untimed first. The median is the middle value of the sorted repeat times.

for _ in range(3):
    a @ b
times = []
for _ in range(15):
    start = time.perf_counter()
    a @ b
    times.append((time.perf_counter() - start) * 1000)
print(f"Median of 15 warm runs: {statistics.median(times):.0f} ms")

Repeats spread out

Even warm repeats disagree with each other. One slow run can come from other work on the machine.

print(f"Fastest warm run: {min(times):.0f} ms")
print(f"Slowest warm run: {max(times):.0f} ms")

Example

import statistics
import time

import torch

torch.manual_seed(0)
a = torch.randn(512, 512)
b = torch.randn(512, 512)

start = time.perf_counter()
c_first = a @ b
naive_ms = (time.perf_counter() - start) * 1000

times = []
result = None
for _ in range(15):
    start = time.perf_counter()
    result = a @ b
    times.append((time.perf_counter() - start) * 1000)
honest_ms = statistics.median(times)

print("Result shape:", c_first.shape)
print("Same result both ways:", torch.equal(c_first, result))
print(f"Naive single run: {naive_ms:.0f} ms")
print(f"Honest median of 15 runs: {honest_ms:.0f} ms")

Key points

  • Warmup: the first untimed runs that let the machine reach its normal speed
  • Use time.perf_counter() to read a clock built for timing short stretches
  • A single cold timing mixes startup costs into the number
  • Median: the middle value when the repeat times are sorted
  • The median of many repeats resists one slow outlier
  • Timing values change from run to run and from machine to machine
  • In short, an honest benchmark warms up, repeats, and reports the median
  • CPU work finishes before the next line runs, so a plain clock is honest here

In this example, two 512 by 512 matrices are multiplied in two ways. The naive timing wraps one single call in a clock. The honest timing repeats the same call 15 times and takes the median. Both ways produce the same result, and the equality print proves the maths never changed. The two times disagree, and a fresh run can flip which one is larger. Both times vary on every run and on every machine. The median is the number worth quoting, because one odd run cannot drag it far.

Note

GPU timing needs extra care, and this course stays on CPU.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why does an honest benchmark repeat the work many times?