Julia Benchmarking Rigor

Benchmarking rigor is a strict method for measuring a program fairly. It warms up the code, repeats each measure, and keeps the smallest count.

Warm up, repeat, and keep the smallest count

Warm-up comes first, then the same call is measured three times. The smallest count is kept.

function total_upto(n)
    total = 0
    for i in 1:n
        total += i
    end
    return total
end
total_upto(10)
a = @allocated total_upto(100)
b = @allocated total_upto(100)
c = @allocated total_upto(100)
println(total_upto(100))
println(min(a, b, c))

Measure a growing build at two sizes

The same building call is measured at 10 entries and at 100 entries. The smallest count at each size shows the growth.

function build_upto(n)
    return collect(1:n)
end
build_upto(3)
a = @allocated build_upto(10)
b = @allocated build_upto(10)
c = @allocated build_upto(100)
d = @allocated build_upto(100)
println(min(a, b))
println(min(c, d))

Compare a flat read with a growing build

A flat read and a growing build are measured at a large size. Only the build grows.

function first_item(xs)
    return xs[1]
end
function build_upto2(n)
    return collect(1:n)
end
small = collect(1:10)
large = collect(1:1000)
first_item(small)
first_item(large)
build_upto2(3)
println(@allocated first_item(large))
println(@allocated build_upto2(100))

Example

function first_item2(xs)
    return xs[1]
end
function build_range(n)
    return collect(1:n)
end
small = collect(1:10)
large = collect(1:1000)
first_item2(small)
first_item2(large)
build_range(3)
a = @allocated first_item2(small)
b = @allocated first_item2(small)
c = @allocated first_item2(large)
d = @allocated first_item2(large)
e = @allocated build_range(10)
f = @allocated build_range(10)
g = @allocated build_range(1000)
h = @allocated build_range(1000)
println(min(a, b))
println(min(c, d))
println(min(e, f))
println(min(g, h))

Key points

Benchmark: a benchmark is a fair measurement made with warm-up, repeats, and scaling.
Warm-up: run code once before measuring, so compilation does not pollute the count.
Repeat: measure the same call more than once, because a single count can wobble.
Smallest: keep the smallest count from the repeats, since it best shows the code with the least outside noise.
Scaling: measure at a small input and a large input, so a flat pattern and a growing pattern show apart.
Fair compare: compare two versions with the same warm-up, repeats, and input sizes.

In this example, a flat read and a growing build are both warmed up. Each is measured twice at a small size and at a large size. The smallest counts are kept. The flat counts stay at 0 while the building counts grow from 144 to 8064.

Note

never use wall-clock times as evidence, because they change from run to run.

Try it Yourself

Edit the code, then press Run.

Running is simulated · JuliaHub

function first_item2(xs)
    return xs[1]
end
function build_range(n)
    return collect(1:n)
end
small = collect(1:10)
large = collect(1:1000)
first_item2(small)
first_item2(large)
build_range(3)
a = @allocated first_item2(small)
b = @allocated first_item2(small)
c = @allocated first_item2(large)
d = @allocated first_item2(large)
e = @allocated build_range(10)
f = @allocated build_range(10)
g = @allocated build_range(1000)
h = @allocated build_range(1000)
println(min(a, b))
println(min(c, d))
println(min(e, f))
println(min(g, h))

Exercise

Why keep the smallest count from repeated measures?

The smallest count is always 0 Repeats make the code run faster It best shows the code with the least outside noise Larger counts are always wrong