{"slug": "zfs-high-availability-in-2026-full-how-to-and-helper-scripts", "title": "ZFS High Availability in 2026 - Full How-to and helper scripts", "summary": "A new open-source how-to and helper-script set delivers high-availability ZFS storage on two commodity hosts and one SAS disk shelf for under $4,000, versus the $25,000-$30,000 commercial HA-ZFS products charge. The setup keeps one ZFS pool imported on exactly one host at a time, using a Pacemaker/corosync two-node cluster, an OCF resource agent (ocf:zfsha:Pool), SATA affiliation handoff through SMP PHY CONTROL, SCSI persistent reservation fencing (zfs-ha-pr), and NFS/SMB failover via the ZFS sharenfs property. The author reports zero NFS cutover gaps longer than 2 seconds for a 1-record-per-second client through a floating VIP, validated with sha256 checks on a 5GB dataset across multiple failovers, and a six-scenario test harness covering clean move, crash, power loss, network partition, stuck export and simultaneous boot.", "body_md": "# \n\n## \n\nMicro Center is THE AI Destination: [https://micro.center/c07d51](https://micro.center/c07d51)\n\nSign up for a FREE 128 gig flash drive at Micro Center Austin: [https://micro.center/e1ae53](https://micro.center/e1ae53)\n\nShop Micro Center Ubiquiti Products: [https://micro.center/60a478](https://micro.center/60a478)\n\nVisit Micro Center News: [https://micro.center/be3f27](https://micro.center/be3f27)\n\n**TL;DR:** Two commodity hosts, one SAS disk shelf, one ZFS pool that is\n\nalways imported on exactly one host at a time. NFS (and SMB!) failover via the ZFS `sharenfs` property. Pacemaker/corosync two-node cluster. Everything is open source and documented \\Uffffffff repo with full how-to, validation record, and test harness: **[TODO GITHUB]**\n\n# \n\nFor this video, in addition to the disk shelf, the two computers in question are just what I had for testing: An AM4 system and a Zimaboard system based on a quadcore Intel Atom. Both of these have LSI 9006-8e controllers in them, attached to the disk shelf.\n\n## \n\nYou want HA storage on the cheap: two used servers, one used disk shelf\n\n(eBay NetApp DS460C, HP D2600, Dell MD1200 all work), ZFS for data\n\nintegrity, and automatic failover when a host dies. Commercial HA-ZFS\n\nproducts charge $25-30k for this; the parts cost under $4k.\n\nThe catch depends on your disks:\n\n- **SATA disks behind a SAS expander** can only be owned by one host at a\n time (SATA Tunneling Protocol affiliation). The shelf enforces it, which\n is actually*good* (hardware split-brain protection), but failover\nrequires actively releasing the affiliation.\n- **SAS disks (or SATA behind good interposers)** have no such exclusivity:\nboth hosts can always issue I/O. You must add your own fencing.\n\nThe repo solves both paths.\n\n## \n\n- **The handoff primitive** : SMP PHY CONTROL operations through the\n expander to release/acquire disk affiliations. Validated end-to-end with\ndata integrity checks (5GB dataset, sha256, across multiple failovers).\n- **An OCF resource agent** (`ocf:zfsha:Pool` ) that does the full sequence:\n rescan ? verify exclusivity evidence ? import on start; unmount NFS ?\nexport ? quiesce ? release on stop.\n- **Pluggable handoff strategies** (`strategies.d/` ):`stp` (SATA\n affiliation dance),`sas` (SCSI persistent reservations),`null`\n (MMP-only with fencing),`auto` (probes the disk and picks).\n- **SCSI persistent reservation fencing** (`zfs-ha-pr` ): per-host stable\n keys, register/reserve/release/preempt. The disk itself enforces single\n ownership: every write from a non-owner gets RESERVATION CONFLICT at\n drive firmware level, independent of both OSes. Validated on a\ndual-controller shelf.\n- **Controlled cutover protocol** (`zfs-ha-cutover` ): for disks with no\n transport exclusivity (plain SATA, no interposers). Ordered\n release/acquire with settle delays, preflight checks on both sides,\n refuses to run if the peer is unreachable or still holds the pool.\nThree consecutive full round-trips verified with data checks.\n- **Boot-time device quiesce** : a udev rule deletes the HA pool’s devices\n the instant they appear at boot, so a booting host never probes the\npeer’s disks (no I/O error spam, no racy re-affiliation).\n- **NFS failover via `sharenfs`** : the ZFS property carries the NFS export\n with the pool import. No separate NFS resource agents. Measured cutover:\nzero gaps > 2s for a 1-record/second NFS client through the floating VIP.\n- **A shelf health checker** (`shelf-health.py` ): SES-based, reports PSU/\n fans/temperature/voltage/slots/IOM status, console + JSON output for\nmonitoring tools.\n- **A test harness** : six cutover scenarios (clean move, crash, power loss,\n network partition, stuck export, simultaneous boot) with a durability-\nverifying write client.\n\n## \n\n1. \n**Stale affiliations after a crash.** If the owning host dies, its\n affiliation persists. The survivor must hard-reset the disk’s phy\n (SMP PHY CONTROL op=2) to release it. A simple clear (op=6) is refused\nwhile the dead host’s phys are still electrically up.\n2. \n**Boot-time probing.** A host that boots while the peer owns the disks\n gets I/O errors and the kernel*drops the disks from discovery* . The fix:\n delete the devices from the OS before anything probes them (udev rule),\nand recover with a host-level reset when promoted.\n3. \n**MMP (multihost) protection.** ZFS’s`multihost=on` detects ambiguous\n ownership and suspends the pool. Great safety net… but a suspended\n import can deadlock both hosts. The handoff procedure sequences around\nthis: export ? quiesce? release? wait for disks to settle? import.\n4. \n**ESTALE vs connection drop.** The VIP must tear down*before* the pool\n stops, or NFS clients get ESTALE errors instead of a clean connection\ndrop that hard mounts retry silently.\n\n## \n\nOn a NetApp DS460C (IOM12) with SATA disks behind interposers and LSI\n\nSAS3008 HBAs, Rocky Linux 10.2, OpenZFS 2.2.11, pacemaker 3.0.1 /\n\ncorosync 3.1.10:\n\n- Clean handoff A?B?A with byte-identical data (sha256)\n- Crash recovery (wedged driver ? phy hard reset ? takeover)\n- Power-loss recovery path (nexus-loss detection)\n- NFS cutover with zero measurable client gaps\n- SCSI persistent reservation fencing: peer writes blocked at drive\nfirmware while owner holds the reservation\n- Controlled cutover: 3 consecutive full round-trips of a 4x3TB raidz1,\ndata markers intact in both directions, ~30-40s per direction\n- Dual-ring corosync (LAN + private subnet), two-node quorum\n- Pacemaker resource failover with VIP colocation\n- Boot-time quiesce (udev rule) on both hosts\n- 5GB dataset integrity across all scenarios\n\nFull validation record with commands and outputs: `docs/VALIDATION.md` in\n\nthe repo. Field notes (failure modes that cost the most time): `docs/ HOWTO.md`\n\n## \n\n- **SATA disks, native** (affiliation enforced by the shelf): the STP\nhandoff machinery solves it completely.\n- **SAS disks / SATA behind good interposers** : no transport exclusivity\n use SCSI persistent reservation fencing (`zfs-ha-pr` ,`fencing=pr` ).\n Verified: the peer’s writes get RESERVATION CONFLICT while the owner\nholds the reservation; reads still pass (type 5 WE-R).\n- **Plain SATA without interposers** (no transport release at all): use\n the controlled cutover protocol (`zfs-ha-cutover` ) for planned\nmaintenance when both hosts are healthy. Not an emergency-failover tool.\n- **Interposer quality matters enormously.** Bad interposers mask the\n drive serial, fabricate SAS addresses, and collapse throughput under\n load (150 MB/s ? under 2 MB/s, constant “Power-on or device reset”\n storms). Good ones pass through the real identity and run at full speed.\n Fast diagnostic: check`sas_address` OUI and serial visibility.\n- **Cabling** : recommend one host per controller. Dual-cabling both hosts\n to both controllers works but forces multipath (each disk appears twice\nper host  a same-disk-twice corruption hazard if you get it wrong).\n- Compatible shelves: anything with SAS expanders and SES  NetApp DS460C,\nHP D2600/D2700/D3600, Dell MD1200, DataON, etc.\n\n## \n\n- **Automatic failover is not wired to the cluster yet.** The cutover\n protocol is proven (3 verified round-trips) but is currently a manual\n command, not a pacemaker resource. The OCF agent exists; wiring it into\na running pacemaker/corosync pair is the remaining step.\n- **Emergency failover with no transport release** (plain SATA, owner\n dead) has no clean answer  without PR support there is nothing to\n preempt. Fence the dead host, then rescan and import; MMP is the\ncorruption backstop.\n- **SBD** requires a shared-block-capable disk (SAS, not SATA-tunneled);\nthe mailbox heartbeat design is stubbed but not yet deployed.\n- **Marginal shelves eat disks.** Four disks dropped off the reference\n shelf during the build (link negotiates, device never enumerates) and\n none were recoverable from the host side  all needed physical reseats.\nBudget for this when buying used hardware.\n\n## \n\nThe repo has a complete deployment guide (`docs/HOWTO.md`), the canonical\n\nhandoff algorithm (`docs/HANDBOFF.md`), and a test harness\n\n(`tests/`). Start with the HOWTO: it’s shelf-agnostic and covers\n\neverything from package installs to the pacemaker resource definitions.\n\nAnd remember mostly the pacemaker, corosync, HA stuff is what RedHat was designed to support. We really aren’t doing anything super special here.\n\nRepo: **[TODO]**\n\nGitHub: [GitHub - wendelltron/zfs-ha: ZFS HA notes and scripts (homelab type context) · GitHub](https://github.com/wendelltron/zfs-ha)\n\nQuestions/corrections welcome – particularly interested in hearing from\n\nanyone who’s run SAS disks (native or interposer) on a shared shelf like we’re doing here. I was sort of disappointed sata-native (no interposer) was so flaky (Proxmox likes to email me about 1-2 SATA command errors a week without the interposers. Harmless, and this only happens under extreme load. One or two transport errors a week is.. tolerable, but annoying. An interposer seems to fix this kind of failure.)\n\nOh, did it increase by one? did it now? Okay…\n\nThese are pretty cheap on ebay:\n\ncombo SAS + IB mellanox/infiniband (10/sometimes 40 gbe)  pretty cheap on ebay", "url": "https://wpnews.pro/news/zfs-high-availability-in-2026-full-how-to-and-helper-scripts", "canonical_source": "https://forum.level1techs.com/t/zfs-high-availability-in-2026-full-how-to-and-helper-scripts/258994#post_8", "published_at": "2026-10-29 17:14:14+00:00", "updated_at": "2026-10-29 17:19:58.372685+00:00", "lang": "en", "topics": ["ai-infrastructure"], "entities": ["ZFS", "Pacemaker", "corosync", "NetApp DS460C", "HP D2600", "Dell MD1200", "ZimaBoard", "Micro Center"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/zfs-high-availability-in-2026-full-how-to-and-helper-scripts", "markdown": "https://wpnews.pro/news/zfs-high-availability-in-2026-full-how-to-and-helper-scripts.md", "text": "https://wpnews.pro/news/zfs-high-availability-in-2026-full-how-to-and-helper-scripts.txt", "jsonld": "https://wpnews.pro/news/zfs-high-availability-in-2026-full-how-to-and-helper-scripts.jsonld"}}