Show HN: Epochnbd – Crash-Consistent Linux NBD Block Device on GCS/Google Drive

A developer released epochnbd, an experimental pure-Go Linux Network Block Device engine that exposes a raw block device backed by Google Cloud Storage, Google Drive, or a local directory, with a CRC32-verified write-ahead log and atomic compare-and-swap epoch checkpoints. The project, built with Gemini Argon and published as a Show HN post, uses 512 KiB chunks on GCS and local storage and 4 MiB chunks on Google Drive, and advances its manifest.json only after every dirty chunk in an epoch uploads successfully via GCS ifGenerationMatch preconditions. The author labels epochnbd a personal hobby project provided "AS IS" without warranty, warning of data loss, corruption, and cloud billing risks and urging independent backups.

Caution Disclaimer — Experimental Hobby Project Use at Your Own Risk epochnbd is a personal hobby project developed with Gemini Argon . It is provided "AS IS" without warranty of any kind. Storage engines and kernel block devices carry inherent risks of bugs, corruption, or cloud billing surprises. Use this software entirely at your own risk —always maintain independent backups of important data. The author assumes no responsibility or liability for any data loss, corruption, system instability, or other damages resulting from its use. epochnbd is a pure-Go CGO ENABLED=0 Linux Network Block Device NBD engine that exposes a true POSIX raw block device /dev/nbdX or TCP NBD export backed by Google Cloud Storage gs:// , Google Drive gdrive:// , or a local directory file:// , with local sparse caching, a CRC32-verified Write-Ahead Log WAL , and atomic Compare-And-Swap CAS epoch checkpoints. Unlike FUSE object-storage mounts gcsfuse , s3fs , epochnbd operates strictly at the raw block layer : - Full POSIX & Database Compatibility: Supports mmap , flock / fcntl locks, fsync flush barriers, SQLite in WAL mode, Git repositories, and standard Linux filesystems ext4 , xfs , btrfs . - Bring Your Own Encryption Zero-Knowledge Cloud Storage : Stack standard Linux LUKS2 dm-crypt , VeraCrypt, or any block/filesystem encryption on top of /dev/nbd0 —or run unencrypted when desired. When encrypted at the block layer, cloud storage only ever sees opaque fixed-size ciphertext chunks 512 KiB default on GCS/LocalFS, 4 MiB default on Google Drive . - Zero Torn Cloud Checkpoints: Dirty chunks are uploaded to immutable, epoch-tagged objects chunks/000042 e7.bin , and manifest.json advances atomically via GCS generation preconditions ifGenerationMatch only after every dirty chunk in the epoch succeeds. Application / SQLite / Git │ ▼ Filesystem ext4 / xfs / btrfs ┐ │ │ Optional user-space / kernel stack Encryption LUKS2 / VeraCrypt / none ┘ Managed automatically if --mount-dir is set │ ▼ Linux Kernel /dev/nbd0 │ NBD CMD READ / WRITE / FLUSH / DISC ▼ ┌─────────────────────────────────────────────────────────────┐ │ epochnbd Daemon │ │ │ │ ┌─────────────────────┐ ┌──────────────────────────┐ │ │ │ Sparse cache.img │◄────►│ CRC32 Write-Ahead Log │ │ │ │ + Presence/Dirty │ │ wal.log + fsync │ │ │ │ Chunk Bitmaps │ └──────────────────────────┘ │ │ └──────────┬──────────┘ │ │ │ │ Debounced Compactor │ cloud-strict │ │ ▼ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ StorageBackend: GCS / Google Drive / LocalFS │ │ │ └───────────────────────────────────────────────────────┘ │ └─────────────────────────────┬───────────────────────────────┘ │ ▼ Cloud Bucket / Drive Folder e.g. gs://my-bucket/vault or gdrive://root/vault ├── manifest.json Atomic CAS pointer + single-writer lease ├── chunks/000000 e1.bin Immutable 512 KiB blocks per epoch ├── chunks/000003 e2.bin └── wal/seq 00000012.seg Optional cloud-strict WAL segments <cache-dir / ├── cache.img Sparse raw disk image apparent size = --size; uses disk only for touched blocks ├── wal.log Append-only binary WAL with per-record IEEE CRC32 checksums ├── state.meta Atomic metadata 0600 : volume uuid, local epoch, present bits, dirty bits ├── epochnbd.pid Active daemon PID while mounted managed by epochnbd-vault.sh, removed on down └── epochnbd.daemon.log Daemon startup, recovery, and epoch compaction logs managed by epochnbd-vault.sh - cache.img Sparse Block Cache : Created via Truncate DiskSize as a Linux sparse file. Untouched regions take 0 bytes on your local SSD. Chunks are faulted in from cloud storage on first read or partial-chunk read-modify-write and served locally thereafter. - wal.log Write-Ahead Log : Stores sequential Write and FlushBarrier records "CNBDWAL1" header before data is compacted to cloud storage. Trimmed automatically once an epoch commits to manifest.json ; torn tail records from power loss are automatically truncated to the last valid CRC32 boundary on startup. - state.meta Persistent Cache State : Atomically written tmp + fsync + rename JSON tracking volume uuid , local epoch , cached per-chunk epochs, PresentBitmap , and DirtyBitmap : - Prevents accidentally attaching a cache directory belonging to a different volume UUID. - Enables warm-cache reuse across down / up cycles: if another machine updated the volume in the cloud, OpenEngine compares state.meta against manifest.json and invalidates only the specific chunks whose epoch changed. - When is it safe to delete <cache-dir ? - Keep <cache-dir across normal down / up sessions for fast warm-cache reads. - Safe to delete anytime after a clean down all dirty chunks are already committed to the cloud; the next up simply starts with a cold sparse cache and faults chunks in on demand . - Do NOT delete after an unclean crash kill -9 or power loss before running up on that machine— wal.log and cache.img hold any uncompacted local writes that Pre-Mount Recovery will replay and push to the cloud on next start. | Object Key | Purpose | |---|---| | manifest.json | Plain JSON volume descriptor: volume uuid , disk size , chunk size , monotonic epoch , last wal seq , chunk epochs map chunkIdx - epoch , and lease holder . Updated only via atomic CAS. | | chunks/%06d e%d.bin | Immutable ChunkSize default 512 KiB for gs:// / file:// , 4 MiB for gdrive:// raw block payload for chunk idx written in epoch e . Unmapped chunk indices are sparse 0x00 . Superseded chunk epochs are garbage-collected after manifest.json advances. | | wal/seq %08d.seg | Only in --sync-mode=cloud-strict Remote WAL delta segments uploaded on every fsync barrier before epoch compaction folds and deletes them. | When using epochnbd-vault.sh with a gs:// backend without a persistent GOOGLE APPLICATION CREDENTIALS key , session OAuth credentials live only in RAM at /run/user/$UID/epochnbd/gcloud-nbd0 or /dev/shm/epochnbd-$UID/... and are revoked + deleted automatically during down . 1. Atomic Epoch Switching & Selective GC: Writes hit local sparse cache.img + wal.log immediately. In the background every 5s of write idle, when wal.log = 32 MiB , on SIGUSR1 , or on graceful shutdown , the compactor snapshots dirty chunks, uploads them in parallel as chunks/<idx e<Epoch+1 .bin , commits manifest.json via atomic CAS, trims compacted WAL records, and garbage-collects only superseded chunk versions. 2. Two Selectable Flush Barrier Modes --sync-mode : - local-wal default : fsync NBD CMD FLUSH persists to local wal.log + cache.img for fast local latency while compacting epochs asynchronously to the cloud. - cloud-strict : Every fsync barrier also uploads a WAL delta segment wal/seq <N .seg to cloud storage before acknowledging the flush to the kernel RPO = 0 even if the local machine is destroyed mid-session . Durability Summary What happens on a crash? : - Process crash or power loss up on the same machine : Zero data loss in both local-wal and cloud-strict modes for all fsync ed writes, because Pre-Mount Recovery replays local wal.log and pushes a new cloud epoch before mounting. - Total machine/SSD destruction recover on a new machine : Rolls back cleanly to the last committed cloud epoch ~5s of write idle in local-wal mode , or zero data loss RPO = 0 in cloud-strict mode. 3. Pre-Mount Crash Recovery: Before opening the NBD listener or attaching /dev/nbdX , OpenEngine checks state.meta , wal.log , and remote wal/seq .seg blobs. Any uncommitted writes from a prior crash kill -9 or power loss are replayed, compacted into a new immutable epoch, and committed via CAS before the block device becomes visible to the OS. 4. Single-Writer Lease & Split-Brain Fencing: manifest.json records lease holder hostname:pid:instance uuid . Same-host restarts automatically reclaim stale crash leases; foreign hosts are blocked unless --force-unlock is explicitly passed recover . If another writer steals the lease mid-session, GCS HTTP 412 Precondition Failed immediately fences the stale node from overwriting manifest.json . 5. Triple Pre-Flight init Protection: init refuses to write a single byte if: - manifest.json already exists at the backend, - any objects exist under the backend prefix chunks/ , wal/ , etc., even if manifest.json was accidentally deleted , or - --cache-dir contains leftover state cache.img , wal.log , state.meta from an existing volume. 6. Colab-Style Per-Session Browser Auth Zero Secret Files : epochnbd-vault.sh authenticates GCS per session via your browser works both on a local desktop and over remote SSH copy-paste , stores the temporary token strictly in RAM tmpfs , and automatically revokes + zero-wipes the token after flushing the final epoch on down . - OS: Linux with the nbd kernel module sudo modprobe nbd - Build: Go 1.22+ builds a static CGO ENABLED=0 binary - System Utilities: util-linux blockdev , lsblk , mount , umount , findmnt - Optional for Managed LUKS2+ext4 Vault Mode : cryptsetup LUKS2 and e2fsprogs mkfs.ext4 , e2fsck - Optional for zero-secret-file GCS browser login : Google Cloud CLI gcloud https://cloud.google.com/sdk/docs/install CGO ENABLED=0 go build -o ./bin/epochnbd ./cmd/epochnbd chmod +x ./epochnbd-vault.sh epochnbd-vault.sh supports two workflows depending on whether --mount-dir is passed: - Managed Vault Mode --mount-dir <dir : Automates LUKS2 aes-xts-plain64 , 512-bit key, Argon2id, 4K sectors + sparse-optimized ext4 -O ^has journal,sparse super2 -E lazy itable init=1,nodiscard -G 64 -m 0 + mount -o noatime,nodiratime,noinit itable so both small 256 MiB and multi-terabyte 2 TiB+ encrypted volumes initialize in seconds without pre-zeroing inode tables into LUKS ciphertext chunks. - Raw NBD Mode omit --mount-dir : Attaches/detaches raw /dev/nbd0 only, leaving formatting, custom encryption e.g. VeraCrypt, ZFS, custom LUKS , or unencrypted filesystems completely up to you. ./epochnbd-vault.sh init \ --bin ./bin/epochnbd \ --backend gs://my-bucket/my-vault \ --cache-dir ~/.local/share/epochnbd/my-vault-cache \ --mount-dir ~/vault \ --size 2G - Opens a Google sign-in flow for this session or prints a URL + verification code prompt automatically over SSH; pass --no-browser anytime to force copy-paste mode . - Initializes manifest.json on GCS, attaches /dev/nbd0 , formats LUKS2 type uppercase YES + passphrase , formats ext4 , mounts ~/vault owned by your user, and checkpoints Epoch 1. - If init is interrupted during interactive LUKS formatting before writing a header, run ./epochnbd-vault.sh resume-init ... with the same flags to finish formatting safely. ./epochnbd-vault.sh down \ --bin ./bin/epochnbd \ --cache-dir ~/.local/share/epochnbd/my-vault-cache \ --mount-dir ~/vault - Syncs and unmounts ~/vault , closes /dev/mapper/epochnbd vault , stops the NBD daemon uploading all remaining dirty chunks and releasing the GCS lease , and revokes + wipes the RAM session token. ./epochnbd-vault.sh up \ --bin ./bin/epochnbd \ --backend gs://my-bucket/my-vault \ --cache-dir ~/.local/share/epochnbd/my-vault-cache \ --mount-dir ~/vault If your previous machine lost power before running down leaving a foreign lease holder in manifest.json , start with an empty --cache-dir on the new host: ./epochnbd-vault.sh recover \ --bin ./bin/epochnbd \ --backend gs://my-bucket/my-vault \ --cache-dir ~/.local/share/epochnbd/my-vault-cache \ --mount-dir ~/vault Omit --mount-dir whenever you want direct control over /dev/nbd0 custom encryption, different filesystems, partitions, or unencrypted volumes : 1. Initialize and attach raw /dev/nbd0 ./epochnbd-vault.sh init \ --bin ./bin/epochnbd \ --backend gs://my-bucket/raw-disk \ --cache-dir ~/.local/share/epochnbd/raw-cache \ --size 4G 2. Use /dev/nbd0 however you like e.g., sparse-optimized ext4, XFS, Btrfs, VeraCrypt, etc. sudo mkfs.ext4 -F -b 4096 -O ^has journal,sparse super2 -E lazy itable init=1,nodiscard -G 64 -m 0 /dev/nbd0 sudo mount -o noatime,nodiratime,noinit itable /dev/nbd0 /mnt/mydisk ... read & write files in /mnt/mydisk ... 3. Unmount/close your upper layers first, then detach & sync to cloud sudo umount /mnt/mydisk ./epochnbd-vault.sh down \ --bin ./bin/epochnbd \ --cache-dir ~/.local/share/epochnbd/raw-cache Safety Check in Raw Mode: down inspects lsblk and /sys/block/nbdX/holders before stopping the daemon. If /dev/nbd0 still has an active mount or an open mapper/encryption container, down refuses to detach until you unmount/close it, then runs blockdev --flushbufs before uploading the final epoch. Both GCSBackend and GDriveBackend share a unified 4-tier Zero-Disk + ADC credential resolver ResolveGoogleClientOptions : 1. Direct In-Memory Access Token EPOCHNBD GOOGLE TOKEN Pass a raw OAuth2 access token in EPOCHNBD GOOGLE TOKEN e.g. from Google Colab google.colab.auth or gcloud auth print-access-token . epochnbd constructs an in-memory StaticTokenSource and immediately unsets EPOCHNBD GOOGLE TOKEN from the process environment—zero files are read or written to disk. 2. In-Memory Credential JSON EPOCHNBD GOOGLE CREDENTIALS JSON Pass authorized-user or service-account JSON directly via EPOCHNBD GOOGLE CREDENTIALS JSON . epochnbd parses the credentials in memory, zeroes the raw byte buffer, and unsets the environment variable immediately. 3. Ephemeral RAM Credential File Auto-Unlinking GOOGLE APPLICATION CREDENTIALS - Per-session browser login gs:// & gdrive:// default via epochnbd-vault.sh : Creates an isolated 0700 directory in RAM /run/user/$UID/epochnbd/... or /dev/shm/... , never touching ~/.config/gcloud and runs gcloud auth application-default login for gs:// or gcloud auth login --enable-gdrive-access --update-adc for gdrive:// . Pass --no-browser over Remote Desktop or SSH to use Google's verification-code page instead of a localhost redirect. - Custom OAuth 2.0 Desktop Client --oauth-client for gcloud -less hosts : Runs a built-in OAuth2 + PKCE flow local loopback or SSH copy-paste requesting GCS or Drive scopes drive.file + drive . - Whenever GOOGLE APPLICATION CREDENTIALS points into /dev/shm/ , /run/user/ , or EPOCHNBD UNLINK CREDS=1 is set, epochnbd loads the refresh token into Go heap RAM, overwrites the file with zeroes, and unlink s it immediately on startup. 4. Standard Disk ADC Fallback with SUDO USER Support Falls back to /home/$SUDO USER/.config/gcloud/application default credentials.json when invoked under sudo , or standard google.FindDefaultCredentials . You can also run the Go binary directly without the helper script for example, with file:///path local backends or TCP NBD listeners : Usage: epochnbd <subcommand flags Subcommands: init Initialize a new volume manifest on the cloud/local backend mount Mount the cloud-backed block device /dev/nbdX and/or TCP NBD server sync Trigger an immediate offline checkpoint and compaction on a volume auth Authorize a per-session GCS OAuth2 token via browser PKCE revoke Revoke and zero-wipe an ephemeral per-session OAuth2 credential file | Flag | Default | Description | |---|---|---| | --backend | required | gs://bucket/prefix , gdrive://<folder id or "root" /subpath , or file:///abs/path | | --cache-dir | required | Local directory for cache.img , wal.log , and state.meta | | --device | "" | Kernel NBD device to attach e.g. /dev/nbd0 , requires root/ CAP SYS ADMIN | | --listen | "" | Serve NBD over TCP or Unix socket e.g. 127.0.0.1:10809 or unix:///tmp/nbd.sock | | --sync-mode | local-wal | local-wal fast local WAL fsync or cloud-strict upload WAL segment on every flush | | --compaction-interval | 5s | Idle debounce duration before dirty chunks compact into a new cloud epoch | | --max-wal-size | 33554432 | Maximum wal.log bytes 32 MiB before forcing immediate compaction | | --max-cache-size | 128M | Maximum local cache.img size before LRU hole punching deallocates clean blocks 0 to disable | | --max-dirty-size | 64M | Maximum uncompacted dirty chunk bytes before applying write backpressure 0 to disable | | --workers | 4 | Concurrent chunk upload workers during epoch compaction | | --force-unlock | false | Force-reclaim a stale lease holder from a crashed foreign host | Send SIGUSR1 to a running mount process at any time to force an immediate cloud epoch upload without unmounting: sudo kill -USR1 "$ sudo cat /path/to/cache/epochnbd.pid " epochnbd enforces conservative default bounds on local cache and uncompacted dirty data: | Option | Default | Purpose | |---|---|---| | --max-cache-size | 128M | Physical host disk cap for cache.img via LRU kernel hole punching 0 to disable | | --max-dirty-size | 64M | Dirty uncompacted chunk threshold before applying writer backpressure 0 to disable | 1. Prevent Host Disk Exhaustion on Massive Cloud Volumes: A virtual disk can be 512 GiB, 2 TiB, or larger. When running on virtual machines, dev containers, or laptops with limited host disk space e.g. 5–15 GB free , copying large file trees or running rsync with unbounded caching would quickly fill the entire host root filesystem, leading to system lockups or forced reboots. 2. Bridge the Disk I/O vs. Cloud Upload Speed Gap: Local applications can write data to Linux block devices at hundreds of megabytes or gigabytes per second, while cloud WAN upload bandwidth especially under Google Drive rate limits is typically much lower. Without dirty backpressure --max-dirty-size 64M , local writes outpace cloud compaction uploads, inflating wal.log and dirty cache indefinitely. Bounding dirty data to 64 MiB throttles writes cleanly to cloud upload throughput. 3. Safe by Default Anywhere: The 128M/64M defaults ensure epochnbd can safely run in low-memory/low-disk environments like VPS or CI containers right out of the box. Cache limits are client-side operational policies , completely independent of the remote volume's authoritative cloud manifest manifest.json . You can change --max-cache-size and --max-dirty-size freely between sessions whenever running up or mount : - Decreasing --max-cache-size e.g. from 1G to 128M : You do not need to delete cache files or run recovery. On startup, the engine scans the existing cache, detects that cached blocks exceed the new lower limit, and automatically executes kernel hole punching fallocate FALLOC FL PUNCH HOLE | FALLOC FL KEEP SIZE on the oldest clean chunks, immediately returning physical disk blocks to the host OS. - Increasing --max-cache-size e.g. from 128M to 2G : Existing cached data remains intact. The cache simply accommodates more clean chunks before LRU eviction triggers. - Disabling Limits: Pass --max-cache-size 0 and --max-dirty-size 0 if you have plenty of local disk and want unlimited local caching. Warning Crash Recovery, Teardown Latency, and the Risks of Unbounded Dirty Cache: 1. Teardown & Unmount Latency: When detaching the volume or shutting down cleanly epochnbd-vault.sh down , the daemon flushes all uncompacted dirty chunks into a final cloud epoch before disconnecting. If --max-dirty-size is set excessively large or 0 for unbounded dirty accumulation , a heavy write burst right before detachment leaves hundreds of megabytes or gigabytes of dirty chunks that must all be uploaded across the network before unmounting and NBD detachment can cleanly complete. 2. Crash Durability vs. Local Cache Loss: By design, the data volume will survive sudden machine crashes, power failure, or unclean detaches kill -9 without data loss—recovering fully upon the next restart— as long as the local cache files cache.img , wal.log , state.meta remain intact on the host . However, in the accident that the host machine fails permanently and its local cache is lost before dirty data is compacted to the cloud, any uncompacted dirty writes will be lost the volume rolls back to the last committed cloud epoch . 3. Post-Crash Re-attachment Latency: Even when local cache files are completely intact after a crash, Pre-Mount Recovery must flush and upload all leftover uncompacted dirty chunks to the cloud before the NBD block device is re-exposed to the OS. If --max-dirty-size was set very large, re-attaching the volume after an unclean shutdown can take a very long time while those gigabytes of dirty chunks are uploaded. Recommendation: Keeping --max-dirty-size bounded e.g. 64M – 512M ensures clean detachment finishes quickly, bounds post-crash re-attachment recovery time, and minimizes dirty data exposed to local drive failure. | Scheme | Example URI | Atomic CAS Mechanism | Status | |---|---|---|---| | gs:// | gs://my-bucket/subdir | GCS Object Generation ifGenerationMatch | Production | | gdrive:// | gdrive://root/sparrow-vault or gdrive://<folderId /vault | Mutex + live fileId:md5Checksum verification trashed = false | Production | | file:// | file:///var/lib/epochnbd-store | flock + SHA-256 verification + fsync + atomic rename | Production | | s3:// | s3://my-bucket/subdir | If-Match / If-None-Match conditional writes | Interface stub roadmap | All unit, race-condition, fault-injection, cold-cache read-modify-write, and mock-GCS/OAuth tests run strictly inside temporary directories t.TempDir without requiring root privileges or cloud credentials: CGO ENABLED=1 go test -v -race ./...