#
#
Micro Center is THE AI Destination: [https://micro.center/c07d51](https://micro.center/c07d51)
Sign up for a FREE 128 gig flash drive at Micro Center Austin: [https://micro.center/e1ae53](https://micro.center/e1ae53)
Shop Micro Center Ubiquiti Products: [https://micro.center/60a478](https://micro.center/60a478)
Visit Micro Center News: [https://micro.center/be3f27](https://micro.center/be3f27)
TL;DR: Two commodity hosts, one SAS disk shelf, one ZFS pool that is
always imported on exactly one host at a time. NFS (and SMB!) failover via the ZFS sharenfs property. Pacemaker/corosync two-node cluster. Everything is open source and documented \Uffffffff repo with full how-to, validation record, and test harness: [TODO GITHUB]
#
For this video, in addition to the disk shelf, the two computers in question are just what I had for testing: An AM4 system and a Zimaboard system based on a quadcore Intel Atom. Both of these have LSI 9006-8e controllers in them, attached to the disk shelf.
#
You want HA storage on the cheap: two used servers, one used disk shelf
(eBay NetApp DS460C, HP D2600, Dell MD1200 all work), ZFS for data
integrity, and automatic failover when a host dies. Commercial HA-ZFS
products charge $25-30k for this; the parts cost under $4k. The catch depends on your disks:
- SATA disks behind a SAS expander can only be owned by one host at a time (SATA Tunneling Protocol affiliation). The shelf enforces it, which is actuallygood (hardware split-brain protection), but failover requires actively releasing the affiliation.
- SAS disks (or SATA behind good interposers) have no such exclusivity: both hosts can always issue I/O. You must add your own fencing.
The repo solves both paths.
#
- The handoff primitive : SMP PHY CONTROL operations through the expander to release/acquire disk affiliations. Validated end-to-end with data integrity checks (5GB dataset, sha256, across multiple failovers).
- An OCF resource agent (
ocf:zfsha:Pool) that does the full sequence: rescan ? verify exclusivity evidence ? import on start; unmount NFS ?
export ? quiesce ? release on stop.
- **Pluggable handoff strategies** (`strategies.d/` ):`stp` (SATA
affiliation dance),sas (SCSI persistent reservations),null
(MMP-only with fencing),`auto` (probes the disk and picks).
- **SCSI persistent reservation fencing** (`zfs-ha-pr` ): per-host stable
keys, register/reserve/release/preempt. The disk itself enforces single ownership: every write from a non-owner gets RESERVATION CONFLICT at drive firmware level, independent of both OSes. Validated on a dual-controller shelf.
- Controlled cutover protocol (
zfs-ha-cutover): for disks with no transport exclusivity (plain SATA, no interposers). Ordered release/acquire with settle delays, preflight checks on both sides, refuses to run if the peer is unreachable or still holds the pool. Three consecutive full round-trips verified with data checks. - Boot-time device quiesce : a udev rule deletes the HA pool’s devices the instant they appear at boot, so a booting host never probes the peer’s disks (no I/O error spam, no racy re-affiliation).
- NFS failover via
sharenfs: the ZFS property carries the NFS export with the pool import. No separate NFS resource agents. Measured cutover: zero gaps > 2s for a 1-record/second NFS client through the floating VIP. - A shelf health checker (
shelf-health.py): SES-based, reports PSU/ fans/temperature/voltage/slots/IOM status, console + JSON output for monitoring tools. - A test harness : six cutover scenarios (clean move, crash, power loss, network partition, stuck export, simultaneous boot) with a durability- verifying write client.
#
Stale affiliations after a crash. If the owning host dies, its affiliation persists. The survivor must hard-reset the disk’s phy
(SMP PHY CONTROL op=2) to release it. A simple clear (op=6) is refused
while the dead host’s phys are still electrically up.
Boot-time probing. A host that boots while the peer owns the disks
gets I/O errors and the kerneldrops the disks from discovery . The fix:
delete the devices from the OS before anything probes them (udev rule),
and recover with a host-level reset when promoted.
3.
MMP (multihost) protection. ZFS’smultihost=on detects ambiguous
ownership and suspends the pool. Great safety net… but a suspended
import can deadlock both hosts. The handoff procedure sequences around
this: export ? quiesce? release? wait for disks to settle? import.
4.
ESTALE vs connection drop. The VIP must tear downbefore the pool
stops, or NFS clients get ESTALE errors instead of a clean connection
drop that hard mounts retry silently.
#
On a NetApp DS460C (IOM12) with SATA disks behind interposers and LSI
SAS3008 HBAs, Rocky Linux 10.2, OpenZFS 2.2.11, pacemaker 3.0.1 /
corosync 3.1.10:
- Clean handoff A?B?A with byte-identical data (sha256)
- Crash recovery (wedged driver ? phy hard reset ? takeover)
- Power-loss recovery path (nexus-loss detection)
- NFS cutover with zero measurable client gaps
- SCSI persistent reservation fencing: peer writes blocked at drive firmware while owner holds the reservation
- Controlled cutover: 3 consecutive full round-trips of a 4x3TB raidz1, data markers intact in both directions, ~30-40s per direction
- Dual-ring corosync (LAN + private subnet), two-node quorum
- Pacemaker resource failover with VIP colocation
- Boot-time quiesce (udev rule) on both hosts
- 5GB dataset integrity across all scenarios
Full validation record with commands and outputs: docs/VALIDATION.md in
the repo. Field notes (failure modes that cost the most time): docs/ HOWTO.md
#
- SATA disks, native (affiliation enforced by the shelf): the STP handoff machinery solves it completely.
- SAS disks / SATA behind good interposers : no transport exclusivity
use SCSI persistent reservation fencing (
zfs-ha-pr,fencing=pr). Verified: the peer’s writes get RESERVATION CONFLICT while the owner
holds the reservation; reads still pass (type 5 WE-R).
- **Plain SATA without interposers** (no transport release at all): use
the controlled cutover protocol (`zfs-ha-cutover` ) for planned
maintenance when both hosts are healthy. Not an emergency-failover tool.
- Interposer quality matters enormously. Bad interposers mask the
drive serial, fabricate SAS addresses, and collapse throughput under
load (150 MB/s ? under 2 MB/s, constant “Power-on or device reset”
storms). Good ones pass through the real identity and run at full speed.
Fast diagnostic: check
sas_addressOUI and serial visibility. - Cabling : recommend one host per controller. Dual-cabling both hosts to both controllers works but forces multipath (each disk appears twice per host a same-disk-twice corruption hazard if you get it wrong).
- Compatible shelves: anything with SAS expanders and SES NetApp DS460C, HP D2600/D2700/D3600, Dell MD1200, DataON, etc.
#
- Automatic failover is not wired to the cluster yet. The cutover protocol is proven (3 verified round-trips) but is currently a manual command, not a pacemaker resource. The OCF agent exists; wiring it into a running pacemaker/corosync pair is the remaining step.
- Emergency failover with no transport release (plain SATA, owner dead) has no clean answer without PR support there is nothing to preempt. Fence the dead host, then rescan and import; MMP is the corruption backstop.
- SBD requires a shared-block-capable disk (SAS, not SATA-tunneled); the mailbox heartbeat design is stubbed but not yet deployed.
- Marginal shelves eat disks. Four disks dropped off the reference shelf during the build (link negotiates, device never enumerates) and none were recoverable from the host side all needed physical reseats. Budget for this when buying used hardware.
#
The repo has a complete deployment guide (docs/HOWTO.md), the canonical
handoff algorithm (docs/HANDBOFF.md), and a test harness
(tests/). Start with the HOWTO: it’s shelf-agnostic and covers
everything from package installs to the pacemaker resource definitions.
And remember mostly the pacemaker, corosync, HA stuff is what RedHat was designed to support. We really aren’t doing anything super special here.
Repo: **[TODO]**
GitHub: [GitHub - wendelltron/zfs-ha: ZFS HA notes and scripts (homelab type context) · GitHub](https://github.com/wendelltron/zfs-ha)
Questions/corrections welcome – particularly interested in hearing from
anyone who’s run SAS disks (native or interposer) on a shared shelf like we’re doing here. I was sort of disappointed sata-native (no interposer) was so flaky (Proxmox likes to email me about 1-2 SATA command errors a week without the interposers. Harmless, and this only happens under extreme load. One or two transport errors a week is.. tolerable, but annoying. An interposer seems to fix this kind of failure.)
Oh, did it increase by one? did it now? Okay…
These are pretty cheap on ebay:
combo SAS + IB mellanox/infiniband (10/sometimes 40 gbe) pretty cheap on ebay