🎓 A1Learner
🔍
0 of 46 lessons completed

Zig Benchmark Methodology

Benchmark by counting is a way to compare two versions of code by counting their operations. The version with fewer operations for the same result is the better version.

Count the baseline

The baseline scans the whole array, so it makes eight comparisons. The target is present, but the baseline never stops early.

const std = @import("std");
pub fn main() !void {
    const out = std.io.getStdOut().writer();
    const data = [_]u32{ 3, 6, 9, 12, 15, 18, 21, 24 };
    const target: u32 = 9;
    var comparisons: u32 = 0;
    var found: usize = 0;
    for (data, 0..) |v, i| {
        comparisons += 1;
        if (v == target) found = i;
    }
    try out.print("{d} {d}\n", .{ comparisons, found });
}

Count the early exit

The early-exit version stops at the target, so it makes only three comparisons. It finds the same index as the baseline.

const std = @import("std");
pub fn main() !void {
    const out = std.io.getStdOut().writer();
    const data = [_]u32{ 3, 6, 9, 12, 15, 18, 21, 24 };
    const target: u32 = 9;
    var comparisons: u32 = 0;
    var found: usize = 0;
    for (data, 0..) |v, i| {
        comparisons += 1;
        if (v == target) {
            found = i;
            break;
        }
    }
    try out.print("{d} {d}\n", .{ comparisons, found });
}

Repeat a fixed number of times

Repeating the work ten times gives a steadier count. Ten passes over eight numbers make eighty additions.

const std = @import("std");
pub fn main() !void {
    const out = std.io.getStdOut().writer();
    const data = [_]u32{ 1, 2, 3, 4, 5, 6, 7, 8 };
    var total: u32 = 0;
    var adds: u32 = 0;
    for (0..10) |_| {
        for (data) |v| {
            total += v;
            adds += 1;
        }
    }
    try out.print("{d} {d}\n", .{ adds, total });
}

Example

const std = @import("std");
pub fn main() !void {
    const out = std.io.getStdOut().writer();
    const data = [_]u32{ 3, 6, 9, 12, 15, 18, 21, 24 };
    const target: u32 = 9;
    var baseline: u32 = 0;
    for (data) |v| {
        baseline += 1;
        if (v == target) {}
    }
    var early: u32 = 0;
    for (data) |v| {
        early += 1;
        if (v == target) break;
    }
    try out.print("baseline {d} early {d}\n", .{ baseline, early });
}

Key points

  • Benchmark by counting: benchmark by counting is comparing code versions by their operation counts instead of by time.
  • Baseline: a baseline is the first version of code that later versions are compared against.
  • Fixed workload: a fixed workload is the same input and the same number of repetitions for every version.
  • Repetition: a repetition is one full run of the work being compared.
  • Operation count: an operation count is the total number of small steps a version performs.
  • Fair comparison: a fair comparison is a comparison where only the code version changes and nothing else does.

Both versions search the same array for the same target. The baseline makes eight comparisons and the early-exit version makes three. Fewer operations for the same result makes the early-exit version the better version.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

What must stay the same when comparing two versions by counting?