# Why ZFS is perfect for agent workloads

> Source: <https://ingresslabs.net/blog/zfs-for-agent-workloads>
> Published: 2026-11-19 20:15:24.799052+00:00

**Recommendation:** Model each agent session as a ZFS dataset and each checkpoint as a snapshot, then fork with `zfs clone` so isolation is structural; cap the global ARC yourself on hypervisor hosts.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

# Why ZFS is perfect for agent workloads

We build Hive, a platform where teams run persistent coding agents. Storage turned out to be one of its hardest problems, and ZFS ended up one of its key components — not a filesystem we happened to have, but the layer the whole agent lifecycle stands on.

Agents need three things from storage: fork an isolated, writable environment from a trusted base in constant time; checkpoint and rewind it constantly; and prove what changed. Most platforms assemble these from separate parts — an overlay here, a snapshot daemon there, a diff format bolted on top. ZFS has all three, and the reason is architectural. They are one design, not a feature list.

## The dataset is the unit of administration

            Every operation an agent platform needs — snapshot, clone, hold,
            quota, send, delegated permission — addresses the same object: the
            dataset. That is what makes the lifecycle composable. A session is
            a dataset. A checkpoint is a snapshot of that dataset. A fork is a
            clone of that snapshot. Replication is a send of that snapshot,
            resumable after interruption. Accounting is the
            `usedby*` property family of the same dataset,
            attributing allocated space per tenant — accurate to the block,
            with the usual caveats that block-size rounding and compression
            mean charges never map one-to-one to bytes written. Nothing to
            integrate, because nothing was ever separate.
          

## Copy-on-write is the mechanism

ZFS is a copy-on-write tree. Every block pointer carries a checksum; writes never overwrite in place; a snapshot is a pinned tree root. A clone is a new dataset whose root references the snapshot's tree, and the first write to any block forks only that block — so isolation is structural rather than a permission bit, and destroying a session frees exactly its blocks.

Forking a session is two commands:

```
zfs snapshot base@verified
zfs clone -o readonly=off base@verified sessions/<agent-id>
```

A zvol is a real block device behind virtio-blk, which is why a guest boots a clone directly, with no FUSE or HTTP translator in the I/O path. Incremental send walks the tree diff between two snapshot roots and emits precisely the changed blocks, compressed. The trade, stated once: the ARC is a global cache with no per-agent partitioning, and on a hypervisor host you cap it yourself.

## The toolbox

The same object model gives the platform a shelf of primitives with no counterpart in object storage, each mapping to an agent operation.

`zfs rollback` resets a dataset to its newest snapshot
            in seconds (`-r` reaches older ones, at the cost of the
            snapshots in between). `zfs promote` flips the
            dependency so a verified checkpoint becomes the new base — clones
            only, chains bottom-up. `zfs bookmark` keeps a send
            anchor without keeping the snapshot, though you cannot clone from
            a bookmark, so branch points keep theirs. `zfs diff`
            reports file-level changes between snapshots of a filesystem
            dataset; zvols are single objects and get nothing.
            `.zfs/snapshot/` exposes old state read-only inside the
            live dataset — add it to your backup excludes.
            `zfs allow` delegates snapshot, clone and rollback to
            an unprivileged tenant on their own subtree; delegate
            `destroy` with care. `zpool checkpoint`
            gives the operator a pool-wide rewind: export, import with
            `--rewind-to-checkpoint`, irreversible, discard
            explicit.
          

These commands carry twenty years of ZFS lineage, and most of the failure modes you will hit are already in the man pages.

## What it looks like in production

In production this shows up as ratios that fall out of the architecture rather than tuning. A disk update for a running agent ships as a 65,536-byte signed delta against a 621 MB base, verified end to end: signature, parent identity, block-pack digest. The delta is one manifest in the same registry as the base image, so the durable copy lives in object storage and the pool holds only what it can re-derive. We tune against our own hosts; treat the numbers as illustrative and the architecture as the claim.

## Where S3 fits

The comparison is architectural. S3's unit is an immutable object in a flat namespace, operated on one key at a time. An agent session is a mutable tree with a history. S3 stores our durable artifacts — the registry behind these deltas sits on it — but it contributes no execution primitives: no fork, no diff, no writable block device. Conditional writes supply optimistic concurrency for the one writer per manifest; Object Lock supplies retention.

S3's genuine advantages follow from the same design: erasure-coded durability across availability zones, reachability from any host, and compliance-mode retention that outlives compromised credentials for the retention period. A ZFS pool is single-host — the largest asymmetry. That leaves a clean split: object storage is where state lives; the pool is where state runs.

## The limits

Linux-only, enforced by the compiler. One host per pool. The ARC is yours to cap. No one garbage-collects the clone graph for you, and send across a trust boundary is a problem we have punted on: two hosts, one pool each, for now.

Perfect is the wrong word; this is the least-wrong substrate we have found. Hive ships on it.
