ZFS High Availability in 2026 - Full How-to and helper scripts

A new open-source how-to and helper-script set delivers high-availability ZFS storage on two commodity hosts and one SAS disk shelf for under $4,000, versus the $25,000-$30,000 commercial HA-ZFS products charge. The setup keeps one ZFS pool imported on exactly one host at a time, using a Pacemaker/corosync two-node cluster, an OCF resource agent (ocf:zfsha:Pool), SATA affiliation handoff through SMP PHY CONTROL, SCSI persistent reservation fencing (zfs-ha-pr), and NFS/SMB failover via the ZFS sharenfs property. The author reports zero NFS cutover gaps longer than 2 seconds for a 1-record-per-second client through a floating VIP, validated with sha256 checks on a 5GB dataset across multiple failovers, and a six-scenario test harness covering clean move, crash, power loss, network partition, stuck export and simultaneous boot.

Micro Center is THE AI Destination: https://micro.center/c07d51 https://micro.center/c07d51 Sign up for a FREE 128 gig flash drive at Micro Center Austin: https://micro.center/e1ae53 https://micro.center/e1ae53 Shop Micro Center Ubiquiti Products: https://micro.center/60a478 https://micro.center/60a478 Visit Micro Center News: https://micro.center/be3f27 https://micro.center/be3f27 TL;DR: Two commodity hosts, one SAS disk shelf, one ZFS pool that is always imported on exactly one host at a time. NFS and SMB failover via the ZFS sharenfs property. Pacemaker/corosync two-node cluster. Everything is open source and documented \Uffffffff repo with full how-to, validation record, and test harness: TODO GITHUB For this video, in addition to the disk shelf, the two computers in question are just what I had for testing: An AM4 system and a Zimaboard system based on a quadcore Intel Atom. Both of these have LSI 9006-8e controllers in them, attached to the disk shelf. You want HA storage on the cheap: two used servers, one used disk shelf eBay NetApp DS460C, HP D2600, Dell MD1200 all work , ZFS for data integrity, and automatic failover when a host dies. Commercial HA-ZFS products charge $25-30k for this; the parts cost under $4k. The catch depends on your disks: - SATA disks behind a SAS expander can only be owned by one host at a time SATA Tunneling Protocol affiliation . The shelf enforces it, which is actually good hardware split-brain protection , but failover requires actively releasing the affiliation. - SAS disks or SATA behind good interposers have no such exclusivity: both hosts can always issue I/O. You must add your own fencing. The repo solves both paths. - The handoff primitive : SMP PHY CONTROL operations through the expander to release/acquire disk affiliations. Validated end-to-end with data integrity checks 5GB dataset, sha256, across multiple failovers . - An OCF resource agent ocf:zfsha:Pool that does the full sequence: rescan ? verify exclusivity evidence ? import on start; unmount NFS ? export ? quiesce ? release on stop. - Pluggable handoff strategies strategies.d/ : stp SATA affiliation dance , sas SCSI persistent reservations , null MMP-only with fencing , auto probes the disk and picks . - SCSI persistent reservation fencing zfs-ha-pr : per-host stable keys, register/reserve/release/preempt. The disk itself enforces single ownership: every write from a non-owner gets RESERVATION CONFLICT at drive firmware level, independent of both OSes. Validated on a dual-controller shelf. - Controlled cutover protocol zfs-ha-cutover : for disks with no transport exclusivity plain SATA, no interposers . Ordered release/acquire with settle delays, preflight checks on both sides, refuses to run if the peer is unreachable or still holds the pool. Three consecutive full round-trips verified with data checks. - Boot-time device quiesce : a udev rule deletes the HA pool’s devices the instant they appear at boot, so a booting host never probes the peer’s disks no I/O error spam, no racy re-affiliation . - NFS failover via sharenfs : the ZFS property carries the NFS export with the pool import. No separate NFS resource agents. Measured cutover: zero gaps 2s for a 1-record/second NFS client through the floating VIP. - A shelf health checker shelf-health.py : SES-based, reports PSU/ fans/temperature/voltage/slots/IOM status, console + JSON output for monitoring tools. - A test harness : six cutover scenarios clean move, crash, power loss, network partition, stuck export, simultaneous boot with a durability- verifying write client. 1. Stale affiliations after a crash. If the owning host dies, its affiliation persists. The survivor must hard-reset the disk’s phy SMP PHY CONTROL op=2 to release it. A simple clear op=6 is refused while the dead host’s phys are still electrically up. 2. Boot-time probing. A host that boots while the peer owns the disks gets I/O errors and the kernel drops the disks from discovery . The fix: delete the devices from the OS before anything probes them udev rule , and recover with a host-level reset when promoted. 3. MMP multihost protection. ZFS’s multihost=on detects ambiguous ownership and suspends the pool. Great safety net… but a suspended import can deadlock both hosts. The handoff procedure sequences around this: export ? quiesce? release? wait for disks to settle? import. 4. ESTALE vs connection drop. The VIP must tear down before the pool stops, or NFS clients get ESTALE errors instead of a clean connection drop that hard mounts retry silently. On a NetApp DS460C IOM12 with SATA disks behind interposers and LSI SAS3008 HBAs, Rocky Linux 10.2, OpenZFS 2.2.11, pacemaker 3.0.1 / corosync 3.1.10: - Clean handoff A?B?A with byte-identical data sha256 - Crash recovery wedged driver ? phy hard reset ? takeover - Power-loss recovery path nexus-loss detection - NFS cutover with zero measurable client gaps - SCSI persistent reservation fencing: peer writes blocked at drive firmware while owner holds the reservation - Controlled cutover: 3 consecutive full round-trips of a 4x3TB raidz1, data markers intact in both directions, ~30-40s per direction - Dual-ring corosync LAN + private subnet , two-node quorum - Pacemaker resource failover with VIP colocation - Boot-time quiesce udev rule on both hosts - 5GB dataset integrity across all scenarios Full validation record with commands and outputs: docs/VALIDATION.md in the repo. Field notes failure modes that cost the most time : docs/ HOWTO.md - SATA disks, native affiliation enforced by the shelf : the STP handoff machinery solves it completely. - SAS disks / SATA behind good interposers : no transport exclusivity use SCSI persistent reservation fencing zfs-ha-pr , fencing=pr . Verified: the peer’s writes get RESERVATION CONFLICT while the owner holds the reservation; reads still pass type 5 WE-R . - Plain SATA without interposers no transport release at all : use the controlled cutover protocol zfs-ha-cutover for planned maintenance when both hosts are healthy. Not an emergency-failover tool. - Interposer quality matters enormously. Bad interposers mask the drive serial, fabricate SAS addresses, and collapse throughput under load 150 MB/s ? under 2 MB/s, constant “Power-on or device reset” storms . Good ones pass through the real identity and run at full speed. Fast diagnostic: check sas address OUI and serial visibility. - Cabling : recommend one host per controller. Dual-cabling both hosts to both controllers works but forces multipath each disk appears twice per host a same-disk-twice corruption hazard if you get it wrong . - Compatible shelves: anything with SAS expanders and SES NetApp DS460C, HP D2600/D2700/D3600, Dell MD1200, DataON, etc. - Automatic failover is not wired to the cluster yet. The cutover protocol is proven 3 verified round-trips but is currently a manual command, not a pacemaker resource. The OCF agent exists; wiring it into a running pacemaker/corosync pair is the remaining step. - Emergency failover with no transport release plain SATA, owner dead has no clean answer without PR support there is nothing to preempt. Fence the dead host, then rescan and import; MMP is the corruption backstop. - SBD requires a shared-block-capable disk SAS, not SATA-tunneled ; the mailbox heartbeat design is stubbed but not yet deployed. - Marginal shelves eat disks. Four disks dropped off the reference shelf during the build link negotiates, device never enumerates and none were recoverable from the host side all needed physical reseats. Budget for this when buying used hardware. The repo has a complete deployment guide docs/HOWTO.md , the canonical handoff algorithm docs/HANDBOFF.md , and a test harness tests/ . Start with the HOWTO: it’s shelf-agnostic and covers everything from package installs to the pacemaker resource definitions. And remember mostly the pacemaker, corosync, HA stuff is what RedHat was designed to support. We really aren’t doing anything super special here. Repo: TODO GitHub: GitHub - wendelltron/zfs-ha: ZFS HA notes and scripts homelab type context · GitHub https://github.com/wendelltron/zfs-ha Questions/corrections welcome – particularly interested in hearing from anyone who’s run SAS disks native or interposer on a shared shelf like we’re doing here. I was sort of disappointed sata-native no interposer was so flaky Proxmox likes to email me about 1-2 SATA command errors a week without the interposers. Harmless, and this only happens under extreme load. One or two transport errors a week is.. tolerable, but annoying. An interposer seems to fix this kind of failure. Oh, did it increase by one? did it now? Okay… These are pretty cheap on ebay: combo SAS + IB mellanox/infiniband 10/sometimes 40 gbe pretty cheap on ebay