{"slug": "custom-zig-test-runner-better-ouput-timing-display-and-support-for-special-tests", "title": "Custom Zig Test Runner, better ouput, timing display, and support for special \"tests:beforeAll\" and \"tests:afterAll\" tests", "summary": "This article describes a custom test runner for Zig 0.16 that provides improved output formatting, timing display, and support for special \"tests:beforeAll\" and \"tests:afterAll\" setup/teardown functions. The runner tracks test pass/fail/skip/leak counts, displays the five slowest tests, and includes a custom panic handler that tracks the current test name. The code also supports filtering tests by name and handles memory leak detection through Zig's testing allocator.", "body_md": "-\n-\nSave karlseguin/c6bea5b35e4e8d26af6f81c22cb5d76b to your computer and use it in GitHub Desktop.\n\n[Learn more about bidirectional Unicode characters](https://github.co/hiddenchars)\n\n| // This is for the Zig 0.16. | |\n| // See https://gist.github.com/karlseguin/c6bea5b35e4e8d26af6f81c22cb5d76b/eb15512d6ae49663fa9df6c7a9725b20dab43edd | |\n| // for a version that workson Zig 0.15.2. | |\n| // See https://gist.github.com/karlseguin/c6bea5b35e4e8d26af6f81c22cb5d76b/1f317ebc9cd09bc50fd5591d09c34255e15d1d85 | |\n| // for a version that workson Zig 0.14.1. | |\n| // in your build.zig, you can specify a custom test runner: | |\n| // const tests = b.addTest(.{ | |\n| // .root_module = $MODULE_BEING_TESTED, | |\n| // .test_runner = .{ .path = b.path(\"test_runner.zig\"), .mode = .simple }, | |\n| // }); | |\n| // in your build.zig, you can specify a custom test runner: | |\n| // const tests = b.addTest(.{ | |\n| // .root_module = $MODULE_BEING_TESTED, | |\n| // .test_runner = .{ .path = b.path(\"test_runner.zig\"), .mode = .simple }, | |\n| // }); | |\n| const std = @import(\"std\"); | |\n| const Io = std.Io; | |\n| const builtin = @import(\"builtin\"); | |\n| const Allocator = std.mem.Allocator; | |\n| const BORDER = \"=\" ** 80; | |\n| // use in custom panic handler | |\n| var current_test: ?[]const u8 = null; | |\n| pub fn main(init: std.process.Init) !void { | |\n| var mem: [8192]u8 = undefined; | |\n| var fba = std.heap.FixedBufferAllocator.init(&mem); | |\n| const allocator = fba.allocator(); | |\n| const env = Env.init(init.environ_map); | |\n| std.testing.io_instance = .init(init.gpa, .{ | |\n| .argv0 = .init(init.minimal.args), | |\n| .environ = init.minimal.environ, | |\n| }); | |\n| defer std.testing.io_instance.deinit(); | |\n| const io = std.testing.io; | |\n| var slowest = SlowTracker.init(allocator, io, 5); | |\n| defer slowest.deinit(); | |\n| var pass: usize = 0; | |\n| var fail: usize = 0; | |\n| var skip: usize = 0; | |\n| var leak: usize = 0; | |\n| Printer.fmt(\"\\r\\x1b[0K\", .{}); // beginning of line and clear to end of line | |\n| for (builtin.test_functions) |t| { | |\n| if (isSetup(t)) { | |\n| t.func() catch |err| { | |\n| Printer.status(.fail, \"\\nsetup \\\"{s}\\\" failed: {}\\n\", .{ t.name, err }); | |\n| return err; | |\n| }; | |\n| } | |\n| } | |\n| for (builtin.test_functions) |t| { | |\n| if (isSetup(t) or isTeardown(t)) { | |\n| continue; | |\n| } | |\n| var status = Status.pass; | |\n| slowest.startTiming(io); | |\n| const is_unnamed_test = isUnnamed(t); | |\n| if (env.filter) |f| { | |\n| if (!is_unnamed_test and std.mem.indexOf(u8, t.name, f) == null) { | |\n| continue; | |\n| } | |\n| } | |\n| const friendly_name = blk: { | |\n| const name = t.name; | |\n| var it = std.mem.splitScalar(u8, name, '.'); | |\n| while (it.next()) |value| { | |\n| if (std.mem.eql(u8, value, \"test\")) { | |\n| const rest = it.rest(); | |\n| break :blk if (rest.len > 0) rest else name; | |\n| } | |\n| } | |\n| break :blk name; | |\n| }; | |\n| current_test = friendly_name; | |\n| std.testing.allocator_instance = .{}; | |\n| const result = t.func(); | |\n| current_test = null; | |\n| const ns_taken = slowest.endTiming(io, friendly_name); | |\n| if (std.testing.allocator_instance.deinit() == .leak) { | |\n| leak += 1; | |\n| Printer.status(.fail, \"\\n{s}\\n\\\"{s}\\\" - Memory Leak\\n{s}\\n\", .{ BORDER, friendly_name, BORDER }); | |\n| } | |\n| if (result) |_| { | |\n| pass += 1; | |\n| } else |err| switch (err) { | |\n| error.SkipZigTest => { | |\n| skip += 1; | |\n| status = .skip; | |\n| }, | |\n| else => { | |\n| status = .fail; | |\n| fail += 1; | |\n| Printer.status(.fail, \"\\n{s}\\n\\\"{s}\\\" - {s}\\n{s}\\n\", .{ BORDER, friendly_name, @errorName(err), BORDER }); | |\n| if (@errorReturnTrace()) |trace| { | |\n| std.debug.dumpErrorReturnTrace(trace); | |\n| } | |\n| if (env.fail_first) { | |\n| break; | |\n| } | |\n| }, | |\n| } | |\n| if (env.verbose) { | |\n| const ms = @as(f64, @floatFromInt(ns_taken)) / 1_000_000.0; | |\n| Printer.status(status, \"{s} ({d:.2}ms)\\n\", .{ friendly_name, ms }); | |\n| } else { | |\n| Printer.status(status, \".\", .{}); | |\n| } | |\n| } | |\n| for (builtin.test_functions) |t| { | |\n| if (isTeardown(t)) { | |\n| t.func() catch |err| { | |\n| Printer.status(.fail, \"\\nteardown \\\"{s}\\\" failed: {}\\n\", .{ t.name, err }); | |\n| return err; | |\n| }; | |\n| } | |\n| } | |\n| const total_tests = pass + fail; | |\n| const status = if (fail == 0) Status.pass else Status.fail; | |\n| Printer.status(status, \"\\n{d} of {d} test{s} passed\\n\", .{ pass, total_tests, if (total_tests != 1) \"s\" else \"\" }); | |\n| if (skip > 0) { | |\n| Printer.status(.skip, \"{d} test{s} skipped\\n\", .{ skip, if (skip != 1) \"s\" else \"\" }); | |\n| } | |\n| if (leak > 0) { | |\n| Printer.status(.fail, \"{d} test{s} leaked\\n\", .{ leak, if (leak != 1) \"s\" else \"\" }); | |\n| } | |\n| Printer.fmt(\"\\n\", .{}); | |\n| try slowest.display(); | |\n| Printer.fmt(\"\\n\", .{}); | |\n| std.process.exit(if (fail == 0) 0 else 1); | |\n| } | |\n| const Printer = struct { | |\n| fn fmt(comptime format: []const u8, args: anytype) void { | |\n| std.debug.print(format, args); | |\n| } | |\n| fn status(s: Status, comptime format: []const u8, args: anytype) void { | |\n| switch (s) { | |\n| .pass => std.debug.print(\"\\x1b[32m\", .{}), | |\n| .fail => std.debug.print(\"\\x1b[31m\", .{}), | |\n| .skip => std.debug.print(\"\\x1b[33m\", .{}), | |\n| else => {}, | |\n| } | |\n| std.debug.print(format ++ \"\\x1b[0m\", args); | |\n| } | |\n| }; | |\n| const Status = enum { | |\n| pass, | |\n| fail, | |\n| skip, | |\n| text, | |\n| }; | |\n| const SlowTracker = struct { | |\n| max: usize, | |\n| slowest: SlowestQueue, | |\n| start: Io.Timestamp, | |\n| allocator: Allocator, | |\n| const SlowestQueue = std.PriorityDequeue(TestInfo, void, compareTiming); | |\n| fn init(allocator: Allocator, io: Io, count: u32) SlowTracker { | |\n| const timestamp = Io.Clock.awake.now(io); | |\n| var slowest: SlowestQueue = .empty; | |\n| slowest.ensureTotalCapacity(allocator, count) catch @panic(\"OOM\"); | |\n| return .{ | |\n| .max = count, | |\n| .start = timestamp, | |\n| .slowest = slowest, | |\n| .allocator = allocator, | |\n| }; | |\n| } | |\n| const TestInfo = struct { | |\n| ns: u64, | |\n| name: []const u8, | |\n| }; | |\n| fn deinit(self: *SlowTracker) void { | |\n| self.slowest.deinit(self.allocator); | |\n| } | |\n| fn startTiming(self: *SlowTracker, io: Io) void { | |\n| self.start = Io.Clock.awake.now(io); | |\n| } | |\n| fn endTiming(self: *SlowTracker, io: Io, test_name: []const u8) u64 { | |\n| const timestamp = Io.Clock.awake.now(io); | |\n| const start = self.start; | |\n| self.start = timestamp; | |\n| const ns: u64 = @intCast(start.durationTo(timestamp).toNanoseconds()); | |\n| var slowest = &self.slowest; | |\n| if (slowest.count() < self.max) { | |\n| // Capacity is fixed to the # of slow tests we want to track | |\n| // If we've tracked fewer tests than this capacity, than always add | |\n| slowest.push(self.allocator, TestInfo{ .ns = ns, .name = test_name }) catch @panic(\"failed to track test timing\"); | |\n| return ns; | |\n| } | |\n| { | |\n| // Optimization to avoid shifting the dequeue for the common case | |\n| // where the test isn't one of our slowest. | |\n| const fastest_of_the_slow = slowest.peekMin() orelse unreachable; | |\n| if (fastest_of_the_slow.ns > ns) { | |\n| // the test was faster than our fastest slow test, don't add | |\n| return ns; | |\n| } | |\n| } | |\n| // the previous fastest of our slow tests, has been pushed off. | |\n| _ = slowest.popMin(); | |\n| slowest.push(self.allocator, TestInfo{ .ns = ns, .name = test_name }) catch @panic(\"failed to track test timing\"); | |\n| return ns; | |\n| } | |\n| fn display(self: *SlowTracker) !void { | |\n| var slowest = self.slowest; | |\n| const count = slowest.count(); | |\n| Printer.fmt(\"Slowest {d} test{s}: \\n\", .{ count, if (count != 1) \"s\" else \"\" }); | |\n| while (slowest.popMin()) |info| { | |\n| const ms = @as(f64, @floatFromInt(info.ns)) / 1_000_000.0; | |\n| Printer.fmt(\" {d:.2}ms\\t{s}\\n\", .{ ms, info.name }); | |\n| } | |\n| } | |\n| fn compareTiming(context: void, a: TestInfo, b: TestInfo) std.math.Order { | |\n| _ = context; | |\n| return std.math.order(a.ns, b.ns); | |\n| } | |\n| }; | |\n| const Env = struct { | |\n| verbose: bool, | |\n| fail_first: bool, | |\n| filter: ?[]const u8, | |\n| fn init(map: *const std.process.Environ.Map) Env { | |\n| return .{ | |\n| .verbose = readEnvBool(map, \"TEST_VERBOSE\", true), | |\n| .fail_first = readEnvBool(map, \"TEST_FAIL_FIRST\", false), | |\n| .filter = readEnv(map, \"TEST_FILTER\"), | |\n| }; | |\n| } | |\n| fn readEnv(map: *const std.process.Environ.Map, key: []const u8) ?[]const u8 { | |\n| return map.get(key); | |\n| } | |\n| fn readEnvBool(map: *const std.process.Environ.Map, key: []const u8, deflt: bool) bool { | |\n| const value = readEnv(map, key) orelse return deflt; | |\n| return std.ascii.eqlIgnoreCase(value, \"true\"); | |\n| } | |\n| }; | |\n| pub const panic = std.debug.FullPanic(struct { | |\n| pub fn panicFn(msg: []const u8, first_trace_addr: ?usize) noreturn { | |\n| if (current_test) |ct| { | |\n| std.debug.print(\"\\x1b[31m{s}\\npanic running \\\"{s}\\\"\\n{s}\\x1b[0m\\n\", .{ BORDER, ct, BORDER }); | |\n| } | |\n| std.debug.defaultPanic(msg, first_trace_addr); | |\n| } | |\n| }.panicFn); | |\n| fn isUnnamed(t: std.builtin.TestFn) bool { | |\n| const marker = \".test_\"; | |\n| const test_name = t.name; | |\n| const index = std.mem.indexOf(u8, test_name, marker) orelse return false; | |\n| _ = std.fmt.parseInt(u32, test_name[index + marker.len ..], 10) catch return false; | |\n| return true; | |\n| } | |\n| fn isSetup(t: std.builtin.TestFn) bool { | |\n| return std.mem.endsWith(u8, t.name, \"tests:beforeAll\"); | |\n| } | |\n| fn isTeardown(t: std.builtin.TestFn) bool { | |\n| return std.mem.endsWith(u8, t.name, \"tests:afterAll\"); | |\n| } |\n\n💥\n\nVery interesting, thanks for sharing!\n\nThe `continue`\n\nfor the `is_unnamed_test`\n\ncase, will discard information about such tests, no?\n\n`pass`\n\n, `fail`\n\nor `leak`\n\nwill not be incremented, since the following block is before them all:\n\n```\n        if (is_unnamed_test) {\n            continue;\n        }\n```\n\nShouldn't a leak be considered has a failure?\n\n[@oliverpool](https://github.com/oliverpool) I think you're right that it's best to leave that out and just report unnamed tests, with their weird names, like any other. It's been updated, cheers.\n\nHey, very interesting, thanks for sharing!\n\nFor anyone reading in the future, I noticed two things in zig 0.13.0 at least:\n\n- To add the test runner you should use\n`b.path(\"test_runner.zig\")`\n\n`std.debug.defaultPanic`\n\nhas been deprecated (I could not find any mention to it anywhere 😅), it should be replaced with`std.debug.panicImpl(error_return_trace, ret_addr, msg);`\n\nYes, having a 0.13 version is nice. For the sake of correctness, it's the opposite though:`std.debug.defaultPanic`\n\nhasn't been deprecated it's new. It was [added about a month ago](https://github.com/ziglang/zig/commit/4f8d244e7ea47a8cdb41496d51961ef4ba3ec2af#diff-c439163aff3333f7964979ad02f98a36c80eadbffa7c1704af2e6945ce326b98R454)\n\nThanks, this is exactly what I was looking for!\n\nOne small change I needed to make for 0.14.0-dev (i haven't tested this on 0.13.0 yet). When adding the `.test_runner`\n\nin your `build.zig`\n\n, you need to use a `Build.LazyPath`\n\n, e.g:\n\n``` js\n    const lib_unit_tests = b.addTest(.{\n        .root_source_file = b.path(\"src/root.zig\"),\n        .target = target,\n        .optimize = optimize,\n        .test_runner = b.path(\"test_runner.zig\"), // use `Build.LazyPath` instead of string literal\n    });\n```\n\nYup, you're right [@xiy](https://github.com/xiy) . I've updated it. Thanks.\n\nWhat's the license on this?\n\n[@SimonMeskens](https://github.com/SimonMeskens) anything you want it to be. But let's say MIT if you work for a place that needs something well-defined.\n\nThat's amazing, thanks! It also helps clarify this downstream project: [https://gist.github.com/nurpax/4afcb6e4ef3f03f0d282f7c462005f12](https://gist.github.com/nurpax/4afcb6e4ef3f03f0d282f7c462005f12)\n\n[@karlseguin](https://github.com/karlseguin)\n\nWow! That looks great! Could you make library out of this for usage within `zig zon`\n\n?\n\nSingle file, seems simpler to copy and paste and adjust as needed.\n\nZig 0.14.1 doesn't seem to have the test_functions field in builtin module anymore, do you have a work aruond\n\n```\n//...\nfor (builtin.test_functions) |t| {\n//......\n```\n\n[@Borwe](https://github.com/Borwe) It still does. Are you sure you're on zig 0.14.1?\n\nYou can see the built-in runner is using it (using the 0.14.1 tag)\n\n[https://github.com/ziglang/zig/blob/7c709f920bbe8f01a25392182e4a3fd02bb95219/lib/compiler/test_runner.zig#L99](https://github.com/ziglang/zig/blob/7c709f920bbe8f01a25392182e4a3fd02bb95219/lib/compiler/test_runner.zig#L99)\n\n[@Borwe]It still does. Are you sure you're on zig 0.14.1?You can see the built-in runner is using it (using the 0.14.1 tag)\n\n[https://github.com/ziglang/zig/blob/7c709f920bbe8f01a25392182e4a3fd02bb95219/lib/compiler/test_runner.zig#L99]\n\nSorry, you are right, I was importing the wrong builtin module.\n\nThe writergate also affects this test runner.\n\n[EDIT 2025/10/04: the version above now works with zig 0.15 - you can ignore the version linked in this comment]\n\nHere is a version working with zig 0.15.1: [https://gist.github.com/jonathanderque/c8dbeafc68c1d45e53f629d3c78331a1](https://gist.github.com/jonathanderque/c8dbeafc68c1d45e53f629d3c78331a1)\n\nI'd be curious to see if there is a better patch: I had to modify more than I initially intended. Overall I feel like I'm still struggling with the Writer API changes.\n\nThanks for the wonderful test runner!\n\nI personally added this line to mine to avoid trying to run on files with no tests so that it doesn't clog the output.\n\n```\npub fn main() !void {\n    if (builtin.test_functions.len < 1) return;\n    // ...\n```\n\n[@karlseguin](https://github.com/karlseguin)\n\nI think the runner is supposed to initialize the environ of testing, so this is missing in `main`\n\n?\n\n```\n    std.testing.environ = init.minimal.environ;\n```\n\nOtherwise there is no way to access envvars from your tests (in 0.16, that is).", "url": "https://wpnews.pro/news/custom-zig-test-runner-better-ouput-timing-display-and-support-for-special-tests", "canonical_source": "https://gist.github.com/karlseguin/c6bea5b35e4e8d26af6f81c22cb5d76b", "published_at": "2023-11-01 03:49:51+00:00", "updated_at": "2026-05-23 19:06:12.542508+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Zig"], "alternates": {"html": "https://wpnews.pro/news/custom-zig-test-runner-better-ouput-timing-display-and-support-for-special-tests", "markdown": "https://wpnews.pro/news/custom-zig-test-runner-better-ouput-timing-display-and-support-for-special-tests.md", "text": "https://wpnews.pro/news/custom-zig-test-runner-better-ouput-timing-display-and-support-for-special-tests.txt", "jsonld": "https://wpnews.pro/news/custom-zig-test-runner-better-ouput-timing-display-and-support-for-special-tests.jsonld"}}