In 2004, setting up NFS across two beige desktop boxes in my room felt
like inviting a thunderstorm indoors. You edited /etc/exports, restarted
portmap and rpcbind, crossed your fingers, and waited for the whole desktop
to freeze the moment someone tripped over an Ethernet cable.
Anyone who grew up in those early Linux days
knows the specific dread of a dead network mount. A terminal tab running ls
would enter an unkillable kernel sleep (D state). Even kill -9 bounced off
it. You either lived with a haunted directory tree or reached for the physical
reset button.
Two decades later, our workloads moved to cloud instances and container
sandboxes, yet the friction of touching files across a wire never really
disappeared. We traded manual NFS exports for rsync loops, background sync
daemons, or sshfs. When you work from the Middle East, where
latency is a daily tax
on every round trip, chatty remote filesystems punish every command you type.
At the end of October, Muayyad Alsadi
(@muayyadalsadi) started building a clean-sheet
answer to this problem called fs-relay. Over the past
week, I had the pleasure of contributing upstream fixes across locking, startup
safety, shutdown, and self-update for today’s v0.2.0 release. This is the
story of how Muayyad’s design makes fs-relay ten times faster than the usual
suspects, and what went into making multi-mount SQLite and atomic self-updates
solid.
What’s in this post:
- Why traditional remote mounts struggle
- How Muayyad made fs-relay 10x faster
- Upstream work toward v0.2.0
- Try fs-relay v0.2.0
Why Traditional Remote Mounts Struggle #
When you want multiple machines or coding agents to share a live workspace without copying gigabytes back and forth, you hit five walls almost immediately:
- The classical user-space copy loop : Moving a file across a network the
textbook way repeats the same five steps for every chunk. With a 4 KiB buffer
and a 1 GiB file, that loop runs over 250,000 times, paying 250,000 syscalls,
250,000 user/kernel context switches, and a trail of scattered, fragmented
memory allocations:
- Allocate memory.
- Receive from the network into memory.
- Write from memory to disk.
- Repeat steps 2 and 3 in either direction (net -> disk ordisk -> net ).
- Free memory.
- Round-trip chatty metadata : A simple
tar -xfor project build issues thousands oflookup,getattr,open,read, andclosecalls. Sending each call as a separate network round trip turns a fast link into molasses. - Root privileges and firewall ports : Classic NFS and SMB expect root daemons, kernel modules, and open ports that are painful to expose across networks.
- Broken advisory locks : Most network mounts either ignore POSIX byte-range locks or fake them locally. Point two processes at the same SQLite database and you get silent corruption.
- No fast-lane for large payloads : Other protocols wrap every small chunk
of a large file inside its own message frame, so framing overhead grows
linearly with file size.
fs-relayhas a dedicatedrawpath for large files that pays a constant overhead regardless of size.
How Muayyad Made fs-relay 10x Faster #
Muayyad designed fs-relay around a strict constraint: five Go packages, zero
CGO, rootless Linux FUSE v3 (fusermount3) on the client, and standard HTTP
and WebSockets on the wire.
Instead of trying to micro-optimize the classical copy loop, fs-relay skips
both the user-space copy cycle and the per-file round trips:
- Zero-copy kernel transfers for large files : Large reads and writes bypass
framing overhead on a raw socket and use Linux kernel zero-copy primitives
(
splice(2)andcopy_file_range(2), as Muayyad detailed inhis article on zero-copy single context switching ). Bytes move between socket and disk inside the kernel without user-space buffer allocation, copying, or a context switch on every 4 KiB block. - 1 MiB write-behind buffering : Contiguous client writes coalesce up to 1 MiB on the mount before flushing across the wire.
- Local read-only opens : Opening a file read-only takes no server file
handle. The client resolves
open,flush, andreleaselocally and reads by inode. - Batch directory and small-file prefetching : Directory listings
(
readdir) return entry attributes inline, then batch-prefetch up to 32 small regular files (<= 64 KiB, up to 1 MiB total) viareadsand up to 32 child directories viareaddirsin one trip. - Server-pushed invalidation : Instead of polling for stale attributes, the
mount listens on one WebSocket or chunked
joinstream for server-pushed cache evictions and 2-second heartbeats.
The framing overhead of reading a 1 GiB file should not be ten times that of a
100 MiB file; the goal is constant cost. Once those bottlenecks (the
250,000-syscall copy loop and the per-file round trips) are out of the way,
fs-relay also keeps tiny files and chatty operations lean by encoding frames
with FlatBuffers for in-place reads with no new allocations (a recurring lesson
for anyone who has watched
serialization overhead dominate Go hot paths).
That is why the repository benchmarks test both scenarios, and the numbers
speak for themselves. On small files, unpacking ansible.tar (27 MiB across
5,868 files) takes 203.56 seconds on s3fs (mounted over rustfs) and 107.89
seconds on fuse-nfs, while fs-relay finishes in 15.47 seconds. On large
files, unpacking Antigravity.tar.gz (500 MiB) takes 36.13 seconds on
fuse-nfs and 21.01 seconds on s3fs, compared to 9.78 seconds on fs-relay.
Even more surprising, running DuckDB queries over Parquet files mounted through
fs-relay beat DuckDB reading the exact same files directly from rustfs over
native S3.
Upstream Work Toward v0.2.0 #
Muayyad’s code and SPEC.md are clear enough that jumping in was easy. The
first contribution
(9071dbd)
proposed dual licensing under MIT and the Waqf Digital Public License
(WaqfDPL-1.0), which Muayyad adopted so the project stays free and open in
perpetuity.
From there, the work covered four practical engineering problems.
1. Server-Authoritative POSIX Locks and SQLite Coherence
To run SQLite across two remote mounts without corruption, both mounts must
agree on byte-range locks (getlk, setlk, setlkw) and flock.
PR #3 added the
server-side lock table in internal/local/lock.go.
A classic POSIX lock belongs to a (PID, inode) pair, which breaks across
machines when two hosts reuse the same PID. In fs-relay, a lock holder is
the mount’s HMAC-signed client ID paired with the kernel lock owner. Because
SQLite’s rollback journal hammers three exact byte offsets at 0x40000000, the
lock table tracks dedicated fast-path counters for the pending byte, the
reserved byte, and the 510-byte shared range, spilling arbitrary ranges onto an
overflow list only when needed.
Getting the lock table working was only half the battle. Enabling a dedicated
go test -race job in CI
(PR #25) and running
multi-writer SQLite stress tests exposed subtle distributed races, fixed
across PR #24,
PR #30,
PR #33, and
PR #34:
- Draining write-behind before unlock : If mount A writes a transaction and
unlocks before its buffered bytes reach the server, mount B acquires the lock
and reads stale bytes.
fs-relay-bindnow drains dirty buffers before sendingF_UNLCKand drops cached attributes and prefetched file bodies the moment a lock grant arrives. - Closing wake-up races : When a writer downgrades to a shared read lock or
a waiter receives
EAGAIN, registering the waiter under the lock mutex ensures coalesced broadcast wake-ups never miss a blocked reader. - Guarding late grants : If a file handle closes while a lock request is still in flight, the client immediately releases the late grant so orphaned locks never wedge the namespace.
We verified this on Linux 6.12 with SQLite 3.46.1 in rollback journal mode:
four writers split across two mounts committed 200 rows concurrently, and
PRAGMA integrity_check returned ok on both mounts. (WAL mode stays out of
scope because it relies on an -shm file mapped into shared memory via mmap
on a single kernel, which cannot be shared across separate machines.)
2. Startup Validation and Single-Server Data Locks
Early on, a typo on the command line or a second fs-relay process pointing at
the same data directory could cause confusing state. Across
PR #16,
PR #17,
PR #18,
PR #20,
PR #29, and
PR #35, we added
non-blocking -token-file secret , default http:// URL schemes,
pre-mount directory validation (-force-existing-mnt), and an exclusive
.fs-relay.lock file lock on the data root hardened against symlink
traversal.
3. Bounded Unmount on Interrupts
Anyone who has pressed Ctrl+C on a FUSE daemon while a shell is still sitting
inside the mounted folder knows how easily shutdown can hang. In
PR #19,
PR #37,
PR #38, and
PR #39, we tightened CLI
argument validation and reworked signal shutdown in fs-relay-bind.
On SIGINT or SIGTERM, fs-relay-bind attempts a clean unmount bounded by a
2-second timeout. If open files or a stalled network flush keep the mount busy,
it falls back to an interruptible lazy detach (fusermount3 -uz or
MNT_DETACH) and exits non-zero with a clear message instead of hanging the
terminal.
4. Two-Phase Atomic Self-Update in v0.2.0
For v0.2.0
(PR #40), we added
update, version, and help subcommands to both fs-relay and
fs-relay-bind, alongside a 12-hour cached background release check
(FS_RELAY_NO_UPDATE_CHECK disables it).
Updating two companion binaries (fs-relay and fs-relay-bind) in place has
sharp edges. If the first binary swaps and the second fails, the machine is
left running mismatched versions. Our updater verifies the release tarball
against the GitHub SHA-256 digest (falling back to checksums.txt), preserves
existing file permissions (such as 0700), stages both replacement binaries in
exclusive temporary files in the target directory, and rolls back the primary
binary automatically if the companion commit fails.
Try fs-relay v0.2.0 #
There is genuine satisfaction in contributing to software that refuses unnecessary layers. You do not need root daemons, kernel modules, or heavyweight cluster coordinators to mount a directory across machines. Standard HTTP, careful FUSE caching, and server-side POSIX locks get the job done in five small packages.
Starting a share, mounting it on a client, and updating both binaries in place takes three commands:
mkdir -p ./data/demo /mnt/demo
fs-relay -listen 127.0.0.1:8080 -data ./data -token-file ./token.txt
fs-relay-bind -url 127.0.0.1:8080 -ns demo -token-file ./token.txt /mnt/demo
fs-relay update
You can grab the v0.2.0 binaries or read the design contract in
Muayyad Alsadi’s fs-relay repository,
or check the summary card on the project page.
Have you tried replacing sshfs or rsync loops with user-space FUSE over
HTTP in your own workflows, or are you still nursing old NFS mounts?
Recommended
Continue reading #
Selected from shared topics, related tags, and the recent archive.