{"slug": "protocol-aware-deterministic-simulation-testing", "title": "Protocol-Aware Deterministic Simulation Testing", "summary": "TigerBeetle detailed its protocol-aware deterministic simulation testing (DST) approach, which asserts invariants at the level of each individual replica rather than only at the database level, as shown by the check `if (replica.status == .recovering_head) assert(fault);`. The company said the method goes beyond black-box distributed-systems testing tools such as Jepsen (generative testing) and Antithesis (deterministic hypervisors) to test safety and liveness invariants. TigerBeetle, a distributed database using Viewstamped Replication (VSR), defines its safety invariant as strict serializability and its liveness invariant as staying responsive while a majority of replicas are online.", "body_md": "# Protocol-Aware Deterministic Simulation Testing\n\nTigerBeetle’s deterministic simulator is protocol-aware, which\nenables us to test safety and liveness invariants not just at the\n*database* level, but also at the level of *each individual\nreplica*.\n\n```\nif (replica.status == .recovering_head) assert(fault);\n```\n\nIn this post, we cover the mechanics, method, and merits of going\n*beyond* traditional, black-box methods of testing distributed\nsystems – generative testing (for example, [Jepsen](https://jepsen.io/)), and deterministic hypervisors\n(for example, [Antithesis](https://antithesis.com/product/what_is_antithesis/))\n– to deeply test safety and liveness invariants using protocol-aware\nDST. If you prefer, watch a\n[talk](https://www.youtube.com/watch?v=F78FeuSIX-A)\nin which I cover this and more.\n\n## [Invariants](#invariants)\n\nTo begin, some background on safety and liveness invariants,\nconsensus protocols, and deterministic simulation testing. For those who\ndon’t require a refresher, feel free to jump to [Protocol-Aware DST](#protocol-aware-dst)!\n\nDistributed systems, i.e. systems with multiple interacting nodes,\nare notoriously hard to get right, as they require developers to reason\nabout concurrent execution on multiple machines, and the state space of\ntheir interleavings is vast. Now, we can attack testing such a system\nfrom multiple angles, but today, let’s start with *invariants*.\nWhile testing your distributed system, it is crucial you identify two\nsets of invariants:\n\n- *Safety* : which means nothing bad ever happens. For example,\ntwo nodes never return different results for the same request.\n- *Liveness* : which means something good eventually happens.\nFor example, a request will eventually be responded to provided enough\nnodes are online.\n\nYour testing must then attempt to ascertain whether your system upholds these safety and liveness invariants.\n\nLet us consider a specific distributed system: a consensus-based system. In this system, a consensus protocol is what turns durability into availability, safely. Put simply, a consensus protocol uses the redundancy in the system to provide fault tolerance, while maintaining the illusion of a single node. At a high level, a consensus protocol guarantees the following:\n\n**Fault Tolerance**\n\nThis entails ensuring that faults like process crashes, network\npartitions, and storage corruptions are tolerated and masked. This is\ntypically achieved via replication, i.e. maintaining multiple copies of\nthe data for redundancy. Algorithms of the [Viewstamped\nReplication](https://www.cs.princeton.edu/courses/archive/fall19/cos418/papers/vr-revisited.pdf) flavor guarantee responsiveness as long as a majority of\nreplicas are online. For example, in a system with 3 replicas (where the\nmajority is 2 replicas), they can tolerate up to 1 fault.\n\nThis is the *liveness* invariant of a consensus protocol.\n\n**Agreement**\n\nThis entails ensuring that the multiple copies of the data in the system are consistent with one another. One way to achieve this is by electing a primary replica. All requests flow through the primary, and the order in which the primary executes operations is the order all backups follow, ensuring data consistency.\n\nThis is the *safety* invariant of a consensus protocol.\n\nTigerBeetle is a distributed database that uses VSR for fault\ntolerance and agreement. Routing all requests through the primary\nguarantees [strict\nserializability](https://jepsen.io/consistency/models/strong-serializable), which is the strongest level of isolation a\ndatabase can guarantee. Simply put, strict serializability posits that\nif one operation completes before another begins, the database must\nreflect that order.\n\nTherefore, we can say that the safety invariant of our distributed database under test is strict serializability, and the liveness invariant is simply the liveness property of VSR, i.e. staying responsive as long as a majority of nodes are online.\n\n## [Testing From The *Outside In*](#testing-from-the-outside-in)\n\n*Outside In*\n\nNow, one way to test these invariants is using black-box generative testing, coupled with fault injection, Jepsen-style. Jepsen tests the system from the outside in, using user-visible APIs, embracing the inherent asynchrony and non-determinism in the system. This involves:\n\n- Generating random inputs to probe the vast state space\n- Subjecting the system to faults\n- Asserting whether the safety and liveness invariants are upheld\n\nLast year, we did do that, and here is an excerpt from our [Jepsen](https://jepsen.io/analyses/tigerbeetle-0.16.11)\nreport:\n\nIntegrating Viewstamped Replication with flexible quorums and protocol-aware recovery does not appear to have compromised the key invariant of Strong Serializability.\n\nThat’s wonderful news! As per our Jepsen evaluation, we were still upholding our guarantees to our users, which is crucial because the financial applications that are typically developed on top of TigerBeetle are developed assuming these strong guarantees. These applications rely on the strict serializability we promise them.\n\nHowever, Jepsen-style generative testing and Antithesis-style\ndeterministic hypervisors test the system from the outside in, via\nuser-visible APIs. What about the invariants that aren’t visible at the\nAPI boundary? For foundational infrastructure, we must do better. We\nmust test not only from the *outside in*… but also from the\n*inside out*.\n\n## [Testing From The *Inside Out*](#testing-from-the-inside-out)\n\n*Inside Out*\n\nTigerBeetle is explicitly designed as a *deterministic*\ndistributed database. The idea of deterministic execution in databases\nisn’t new: [FoundationDB](https://www.foundationdb.org/files/fdb-paper.pdf)\nand [Dropbox’s\nSync Engine](https://dropbox.tech/infrastructure/-testing-our-new-sync-engine) are known to have taken advantage of determinism in\ntheir design, alongside deterministic simulation testing.\n\nSo, like FoundationDB, TigerBeetle has *logical* determinism,\nwhere all database code is deterministic and multithreaded concurrency\nis avoided in the control plane.\n\nHowever, we go further and also ensure *physical* determinism,\nwhere replicas in a TigerBeetle cluster converge to a byte-by-byte\nidentical state. In other words, across all the replicas in the cluster,\nthe same physical location corresponds to the same content.\n\nThis logical *and* physical determinism makes TigerBeetle\namenable to Deterministic Simulation Testing, which helps us run the\n*real* consensus and storage engine code in a simulator, which we\ncall the [VOPR](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/docs/internals/vopr.md).\nInside the VOPR, non-deterministic, physical interactions like storage\nand network are replaced with controllable versions, and time is\nsimulated. This allows us to run our distributed database on a single\nmachine, in a single process, with time sped up by orders of\nmagnitude.\n\nWe generate random scenarios to explore the state space of the database: network partitions, disk corruptions, and replica crashes all in the presence of concurrent client operations, and test for various safety and liveness invariants throughout.\n\nBecause time is simulated, exploring scenarios that would take months to encounter in production takes mere minutes. This means that you can quickly find subtle interleavings of events where bugs typically like to hide, and once a bug is found, it can be reproduced over and over again deterministically. Therefore, DST allows developers to test and debug faster, which in turn means that they can build faster.\n\n## [Protocol-Aware DST](#protocol-aware-dst)\n\nBut today, I want to talk about a slightly different aspect of DST.\nWe use DST to test TigerBeetle from the *inside out*, which means\nthat it has complete visibility into each replica’s consensus and\nstorage-level state. With this protocol-awareness, we can peek under the\nhood of the system and start asking it the hard questions.\n\n## [Checking Deep Safety Invariants](#checking-deep-safety-invariants)\n\nWe can deeply check safety invariants not just at the *database\nlevel*, but also across *consensus* and *storage*.\nTesting these safety invariants is crucial because safety is\nhierarchical. Violation of a safety invariant in consensus or storage\ncan ultimately lead to the database violating its core safety invariant\nof strict serializability.\n\nBecause of this, we enforce these safety invariants both in\nproduction *and* in testing. We run with assertions enabled in\nproduction that simply crash the replica if a violation is detected, [downgrading a\ncatastrophic safety violation to an availability violation](https://www.youtube.com/watch?v=wLdDW3u8eww). However,\ngathering global, cluster-level context in production can be expensive,\nso we leave these expensive checks to our protocol-aware DST.\n\n**Consensus Safety**\n\nFor the consensus protocol, the safety invariant is\n*agreement*, i.e. making sure that requests are committed through\nthe write-ahead-log (WAL) in the same order across all replicas.\n\nIn production, we validate the consistency of the WAL when a backup\nreceives a [commit\nmessage](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/src/vsr/replica.zig#L2417-L2429) from the primary. If we observe that a backup’s WAL diverges\nfrom the primary’s, we crash the backup, as this means we’ve violated\nthe protocol’s safety invariant. This is something that should never\nhappen, so we downgrade a safety violation into unavailability.\n\n``` js\nfn on_commit(self: *Replica, message: *const Message.Commit) void {\n    assert(message.header.command == .commit);\n    if (self.journal.header_with_op(message.header.commit)) |commit_entry| {\n        if (commit_entry.checksum == message.header.commit_checksum) {\n            log.debug(\"{}: on_commit: checksum verified\", .{self.log_prefix()});\n        } else if (self.valid_hash_chain_between(message.header.commit, self.op)) {\n            @panic(\"commit checksum verification failed\");\n        }\n```\n\nHowever, since this is driven by a periodic commit message received from the primary, it does not give us the opportunity to check consistency for every single request.\n\nWith protocol-aware DST, we validate this invariant even more deeply.\nEvery time a new request is committed on a replica, our simulator [asserts](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/src/testing/cluster/state_checker.zig#L205-L213)\nthat if that request was committed on another replica, their checksums\nmust match. Effectively, we check that every committed request in the\nWAL is consistent across all replicas.\n\n```\n    assert((commit_a == commit_b) == (checksum_a == checksum_b));\n```\n\n**Storage Safety**\n\nFor the storage engine, the safety invariant is storage determinism – replicas in the TigerBeetle cluster must converge to a byte-by-byte identical state. This means that the LSM trees on each replica run their subprotocols like compaction in exactly the same way, producing exactly the same data on disk. Physical storage determinism lends itself to a better operator experience in production – allowing faster distributed recovery and online verification of the replicas’ state.\n\nAt runtime in production, we *logically* check whether our\nstorage is deterministic or not, using the primary replica’s\n`checkpoint_id` (the checksum over the references to our\nindex data structures). If we observe that the backup’s\n`checkpoint_id` [diverges](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/src/vsr/replica.zig#L2172-L2194)\nfrom the primary’s, we crash the backup, downgrading a safety violation\nto unavailability.\n\n```\nif (message.header.checkpoint_id != self.superblock.working.checkpoint_id() and\n    message.header.checkpoint_id !=\n        self.superblock.working.vsr_state.checkpoint.parent_checkpoint_id)\n{\n    @panic(\"checkpoint diverged\");\n}\n```\n\nWith protocol-aware DST, we take this logical check further by\ndigging deeper into these indexes. For example, for the Manifest, which\nis an index over the on-disk LSM tree, we checksum the metadata of all\ntables, across all levels, and [assert](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/src/testing/cluster/manifest_checker.zig#L40-L46)\nthat the structure of the tree is completely consistent across all\nreplicas.\n\n``` js\nfn manifest_levels_checksum(forest: *const Forest) u128 {\n    var checksum_stream = vsr.ChecksumStream.init();\n    for (0..constants.lsm_levels) |level| {\n        checksum_stream.add(std.mem.asBytes(&level));\n\n        inline for (Forest.tree_id_range.min..Forest.tree_id_range.max + 1) |tree_id_u16| {\n            const tree_id: Forest.TreeID = @enumFromInt(tree_id_u16);\n            const tree_level = forest.tree_for_id_const(tree_id).manifest.levels[level];\n            var tree_tables = tree_level.tables.iterator_from_index(0, .ascending);\n\n            checksum_stream.add(std.mem.asBytes(&tree_id));\n            checksum_stream.add(std.mem.asBytes(&tree_level.table_count_visible));\n            while (tree_tables.next()) |tree_table| {\n                checksum_stream.add(std.mem.asBytes(&tree_table.encode(.{\n                    .tree_id = tree_id_u16,\n                    .event = .insert, // (Placeholder event).\n                    .level = @intCast(level),\n                })));\n            }\n        }\n    }\n    return checksum_stream.checksum();\n}\n```\n\nFinally, with protocol-aware DST, we also *physically* check\nwhether our storage is deterministic. We walk through all of these\nindexes, checksum the actual data (the superblock, the grid, the client\nreplies), and assert that replicas are byte-by-byte identical.\n\n```\ncheckpoint.put(\n    .superblock_checkpoint,\n    vsr.checksum(std.mem.asBytes(&superblock.working.vsr_state.checkpoint)),\n);\ncheckpoint.put(.client_replies, checker.checksum_client_replies(superblock));\ncheckpoint.put(.grid, checker.checksum_grid(Forest, forest, .free_set_from_disk));\n```\n\n## [Checking Deep Liveness Invariants](#checking-deep-liveness-invariants)\n\nRecall that our system’s liveness invariant was to stay responsive to\nuser requests as long as a majority of replicas are online. However,\nsimply testing this *system-level availability* is not enough… A\nmajority of replicas could provide you with the illusion of\navailability: they’re enough to respond to requests, but the rest may be\nstuck!\n\nWith protocol-aware DST, instead of defining liveness invariants for\nthe system as a whole, you can start to define stricter liveness\ninvariants and enforce them for each individual replica. In other words,\nyou can check if replicas are converting durability into availability as\n*efficiently* as they can, while ensuring total order.\n\n**Local Durability To Availability**\n\nSo, we check that replicas efficiently use their *local\ndurability*. For example, when a replica crashes and restarts,\nsometimes it needs to coordinate with the cluster to recover.\nSpecifically, it needs to coordinate if there is corruption at the head\nof its WAL. But if it has enough local durability to recover on its own,\nit *shouldn’t* coordinate.\n\nSo, we [assert](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/src/vopr.zig#L1647-L1650)\nthat if there are no corruptions in the WAL, replicas should\n*never* wind up in a state where they need to coordinate. And\nthis is crucial, because imagine a scenario where all replicas crash and\nrestart together, and then unnecessarily coordinate. The cluster would\nthen be stuck; that’s a serious liveness issue!\n\n```\nif (replica.status == .recovering_head) {\n    // Even with faults disabled, a replica may wind up in\n    // status=recovering_head, in case of a header-prepare view mismatch.\n    assert(fault or header_prepare_view_mismatch);\n}\n```\n\n**Global Durability To Availability**\n\nWe also check that replicas efficiently use the cluster’s *global\ndurability*. Take for instance a scenario where each replica is\nmissing 6 out of 9 blocks, but collectively, all blocks exist somewhere\nin the cluster.\n\nThis is a scenario where most consensus protocols, including the\noriginal VSR, [typically](https://etcd.io/docs/v3.5/op-guide/data_corruption/)\n[assume](https://discuss.hashicorp.com/t/how-to-recover-from-raft-db-corrupt-except-deleting-it/47550)\nthat recovery happens by copying the full state over from an intact\nreplica. But TigerBeetle’s protocol does things a bit differently. We’re\nable to recover all copies, even if there’s only 1 remaining copy of the\nblock in the cluster. Each replica takes advantage of the\n*physical* storage determinism and fetches *only*\nindividual, intact blocks from the other replicas, as opposed to copying\nover the entire state.\n\nAnd with protocol-aware DST we can test that replicas can recover in\nthese scenarios! In other words, we can [assert](https://github.com/tigerbeetle/tigerbeetle/blob/0.17.2/src/vopr.zig#L1235-L1267)\nthat each replica repairs its missing blocks from the intact copies on\nother replicas. If any blocks do remain missing, they must be missing on\n*all* replicas.\n\n```\nwhile (blocks_missing.next()) | block_missing | {\n    for (simulator.cluster.replicas) |replica| {\n        const storage = &simulator.cluster.storages[replica.replica];\n\n        if (storage.area_faulty(.{\n            .grid = .{ .address = block_missing.address },\n        })) continue;\n\n        const block = storage.grid_block(block_missing.address) orelse continue;\n        const block_header = schema.header_from_block(block);\n        if (block_header.checksum == block_missing.checksum) {\n            @panic(\"block found\");\n        }\n    }\n}\n```\n\nAgain, we’re only able to deeply test these safety and liveness invariants because of the protocol awareness of our DST. This is not just executing deterministically (like a hypervisor) and finding violations of system-level invariants (like Jepsen).\n\nInstead, our DST actually understands the protocol itself, with visibility into each replica’s storage and consensus-level state. Again, for foundational infrastructure, we must test not just from the outside in, but also from the inside out.\n\nWith this visibility, you can do more than just deeply test safety and liveness invariants. For example:\n\n- Testing and debugging precise, hand-crafted scenarios. You can ask ‘what happens if…’ questions of your protocol in 30-40 lines of code and get a definitive answer within milliseconds.\n- Benchmarking protocol-level optimizations reliably, without the variance or complexity associated with benchmarking on real hardware.\n\nThese are covered in my\n[talk](https://www.youtube.com/watch?v=F78FeuSIX-A)\nat BugBash ’26, should you wish to dig deeper!", "url": "https://wpnews.pro/news/protocol-aware-deterministic-simulation-testing", "canonical_source": "https://tigerbeetle.com/blog/2026-08-20-protocol-aware-dst/", "published_at": "2026-09-20 04:10:19+00:00", "updated_at": "2026-09-20 04:22:56.115303+00:00", "lang": "en", "topics": ["ai-research"], "entities": ["TigerBeetle", "Jepsen", "Antithesis", "Viewstamped Replication"], "alternates": {"html": "https://wpnews.pro/news/protocol-aware-deterministic-simulation-testing", "markdown": "https://wpnews.pro/news/protocol-aware-deterministic-simulation-testing.md", "text": "https://wpnews.pro/news/protocol-aware-deterministic-simulation-testing.txt", "jsonld": "https://wpnews.pro/news/protocol-aware-deterministic-simulation-testing.jsonld"}}