Show HN: Epochnbd – Crash-Consistent Linux NBD Block Device on GCS/Google Drive

Caution

Disclaimer — Experimental Hobby Project (Use at Your Own Risk)

epochnbd is a personal hobby project developed with Gemini Argon. It is provided "AS IS" without warranty of any kind. Storage engines and kernel block devices carry inherent risks of bugs, corruption, or cloud billing surprises. Use this software entirely at your own risk—always maintain independent backups of important data. The author assumes no responsibility or liability for any data loss, corruption, system instability, or other damages resulting from its use.

epochnbd is a pure-Go ( CGO_ENABLED=0) Linux Network Block Device (NBD) engine that exposes a true POSIX raw block device (/dev/nbdX or TCP NBD export) backed by Google Cloud Storage (gs://), Google Drive ( gdrive://), or a local directory ( file://), with local sparse caching, a CRC32-verified Write-Ahead Log (WAL), and atomic Compare-And-Swap (CAS) epoch checkpoints.

Unlike FUSE object-storage mounts (gcsfuse, s3fs), epochnbd operates strictly at the raw block layer:

• Full POSIX & Database Compatibility: Supportsmmap ,flock /fcntl locks,fsync flush barriers, SQLite in WAL mode, Git repositories, and standard Linux filesystems (ext4 ,xfs ,btrfs ).
• Bring Your Own Encryption (Zero-Knowledge Cloud Storage): Stack standard LinuxLUKS2 (dm-crypt) , VeraCrypt, or any block/filesystem encryption on top of/dev/nbd0 —or run unencrypted when desired. When encrypted at the block layer, cloud storage only ever sees opaque fixed-size ciphertext chunks (512 KiB default on GCS/LocalFS,4 MiB default on Google Drive).
• Zero Torn Cloud Checkpoints: Dirty chunks are uploaded to immutable, epoch-tagged objects (chunks/000042_e7.bin ), andmanifest.json advancesatomically via GCS generation preconditions (ifGenerationMatch ) only after every dirty chunk in the epoch succeeds.

  Application / SQLite / Git
              │
              ▼
   Filesystem (ext4 / xfs / btrfs)        ┐
              │                           │ Optional user-space / kernel stack
   Encryption (LUKS2 / VeraCrypt / none)  ┘ (Managed automatically if --mount-dir is set)
              │
              ▼
      Linux Kernel /dev/nbd0
              │  (NBD_CMD_READ / WRITE / FLUSH / DISC)
              ▼
┌─────────────────────────────────────────────────────────────┐
│                     epochnbd Daemon                         │
│                                                             │
│  ┌─────────────────────┐      ┌──────────────────────────┐  │
│  │  Sparse cache.img   │◄────►│  CRC32 Write-Ahead Log   │  │
│  │ + Presence/Dirty    │      │  (wal.log + fsync)       │  │
│  │   Chunk Bitmaps     │      └──────────────────────────┘  │
│  └──────────┬──────────┘                   │                │
│             │ Debounced Compactor          │ (cloud-strict) │
│             ▼                              ▼                │
│  ┌───────────────────────────────────────────────────────┐  │
│  │ StorageBackend: GCS / Google Drive / LocalFS          │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
     Cloud Bucket / Drive Folder (e.g. gs://my-bucket/vault or gdrive://root/vault)
         ├── manifest.json             (Atomic CAS pointer + single-writer lease)
         ├── chunks/000000_e1.bin      (Immutable 512 KiB blocks per epoch)
         ├── chunks/000003_e2.bin
         └── wal/seq_00000012.seg      (Optional cloud-strict WAL segments)
<cache-dir>/
├── cache.img            # Sparse raw disk image (apparent size = --size; uses disk only for touched blocks)
├── wal.log              # Append-only binary WAL with per-record IEEE CRC32 checksums
├── state.meta           # Atomic metadata (0600): volume_uuid, local_epoch, present_bits, dirty_bits
├── epochnbd.pid         # Active daemon PID while mounted (managed by epochnbd-vault.sh, removed on down)
└── epochnbd.daemon.log  # Daemon startup, recovery, and epoch compaction logs (managed by epochnbd-vault.sh)

• cache.img (Sparse Block Cache): Created viaTruncate(DiskSize) as a Linux sparse file. Untouched regions take0 bytes on your local SSD. Chunks are faulted in from cloud storage on first read (or partial-chunk read-modify-write) and served locally thereafter.
• wal.log (Write-Ahead Log): Stores sequentialWrite andFlushBarrier records ("CNBDWAL1" header) before data is compacted to cloud storage. Trimmed automatically once an epoch commits tomanifest.json ; torn tail records from power loss are automatically truncated to the last valid CRC32 boundary on startup.
• state.meta (Persistent Cache State): Atomically written (tmp +fsync +rename ) JSON trackingvolume_uuid ,local_epoch , cached per-chunk epochs,PresentBitmap , andDirtyBitmap :
  • Prevents accidentally attaching a cache directory belonging to a different volume UUID.
  • Enables warm-cache reuse acrossdown /up cycles: if another machine updated the volume in the cloud,OpenEngine comparesstate.meta againstmanifest.json and invalidatesonly the specific chunks whose epoch changed.
• When is it safe to delete <cache-dir>? - Keep <cache-dir> across normal down /up sessions for fast warm-cache reads.
  • Safe to delete anytimeafter a cleandown (all dirty chunks are already committed to the cloud; the nextup simply starts with a cold sparse cache and faults chunks in on demand).
  • Do NOT delete after an unclean crash (kill -9 or power loss)before runningup on that machine—wal.log andcache.img hold any uncompacted local writes that Pre-Mount Recovery will replay and push to the cloud on next start.

Object Key	Purpose
manifest.json	Plain JSON volume descriptor: volume_uuid ,disk_size ,chunk_size , monotonicepoch ,last_wal_seq ,chunk_epochs map (chunkIdx -> epoch ), andlease_holder . Updatedonly via atomic CAS.
chunks/%06d_e%d.bin	Immutable ChunkSize (default512 KiB forgs:// /file:// ,4 MiB forgdrive:// ) raw block payload for chunkidx written in epoche . Unmapped chunk indices are sparse (0x00 ). Superseded chunk epochs are garbage-collected aftermanifest.json advances.
wal/seq_%08d.seg	(Only in --sync-mode=cloud-strict) Remote WAL delta segments uploaded on every fsync barrier before epoch compaction folds and deletes them.

When using epochnbd-vault.sh with a gs:// backend (without a persistent GOOGLE_APPLICATION_CREDENTIALS key), session OAuth credentials live only in RAM at /run/user/$UID/epochnbd/gcloud-nbd0 (or /dev/shm/epochnbd-$UID/...) and are revoked + deleted automatically during down.

Atomic Epoch Switching & Selective GC: Writes hit local sparsecache.img +wal.log immediately. In the background (every5s of write idle, whenwal.log >= 32 MiB , onSIGUSR1 , or on graceful shutdown), the compactor snapshots dirty chunks, uploads them in parallel aschunks/<idx>_e<Epoch+1>.bin , commitsmanifest.json via atomic CAS, trims compacted WAL records, and garbage-collects only superseded chunk versions. 2. Two Selectable Flush Barrier Modes (--sync-mode):

• local-wal(default) : fsync (NBD_CMD_FLUSH ) persists to localwal.log +cache.img for fast local latency while compacting epochs asynchronously to the cloud.
• cloud-strict : Everyfsync barrier also uploads a WAL delta segment (wal/seq_<N>.seg ) to cloud storage before acknowledging the flush to the kernel (RPO = 0 even if the local machine is destroyed mid-session). Durability Summary (What happens on a crash?):
• **Process crash or power loss (up on the same machine):**Zero data loss in bothlocal-wal andcloud-strict modes for allfsync ed writes, because Pre-Mount Recovery replays localwal.log and pushes a new cloud epoch before mounting.
• Total machine/SSD destruction (recover on a new machine): Rolls back cleanly to the last committed cloud epoch (~5s of write idle inlocal-wal mode), orzero data loss (RPO = 0) in cloud-strict mode.

Pre-Mount Crash Recovery: Before opening the NBD listener or attaching/dev/nbdX ,OpenEngine checksstate.meta ,wal.log , and remotewal/seq_*.seg blobs. Any uncommitted writes from a prior crash (kill -9 or power loss) are replayed, compacted into a new immutable epoch, and committed via CASbefore the block device becomes visible to the OS. 4. Single-Writer Lease & Split-Brain Fencing:manifest.json recordslease_holder (hostname:pid:instance_uuid ). Same-host restarts automatically reclaim stale crash leases; foreign hosts are blocked unless--force-unlock is explicitly passed (recover ). If another writer steals the lease mid-session, GCS HTTP412 Precondition Failed immediately fences the stale node from overwritingmanifest.json . 5. Triple Pre-Flight init Protection:init refuses to write a single byte if:

• manifest.json already exists at the backend,
• any objects exist under the backend prefix (chunks/ ,wal/ , etc., even ifmanifest.json was accidentally deleted), or
• --cache-dir contains leftover state (cache.img ,wal.log ,state.meta ) from an existing volume.

Colab-Style Per-Session Browser Auth (Zero Secret Files):epochnbd-vault.sh authenticates GCS per session via your browser (works both on a** local desktop** and overremote SSH copy-paste), stores the temporary token strictly inRAM (tmpfs) , and automaticallyrevokes + zero-wipes the token after flushing the final epoch ondown .

• OS: Linux with thenbd kernel module (sudo modprobe nbd )
• Build: Go1.22+ (builds a staticCGO_ENABLED=0 binary)
• System Utilities:util-linux (blockdev ,lsblk ,mount ,umount ,findmnt )
• Optional (for Managed LUKS2+ext4 Vault Mode):cryptsetup (LUKS2) ande2fsprogs (mkfs.ext4 ,e2fsck )
• Optional (for zero-secret-file GCS browser login):Google Cloud CLI (gcloud)

CGO_ENABLED=0 go build -o ./bin/epochnbd ./cmd/epochnbd
chmod +x ./epochnbd-vault.sh

epochnbd-vault.sh supports two workflows depending on whether --mount-dir is passed:

• Managed Vault Mode (--mount-dir <dir>): Automates LUKS2 ( aes-xts-plain64 , 512-bit key, Argon2id, 4K sectors) + sparse-optimizedext4 (-O ^has_journal,sparse_super2 -E lazy_itable_init=1,nodiscard -G 64 -m 0 ) + mount (-o noatime,nodiratime,noinit_itable ) so both small (256 MiB ) and multi-terabyte (2 TiB+ ) encrypted volumes initialize in seconds without pre-zeroing inode tables into LUKS ciphertext chunks.
• Raw NBD Mode (omit --mount-dir): Attaches/detaches raw/dev/nbd0 only, leaving formatting, custom encryption (e.g. VeraCrypt, ZFS, custom LUKS), or unencrypted filesystems completely up to you.

./epochnbd-vault.sh init \
  --bin ./bin/epochnbd \
  --backend gs://my-bucket/my-vault \
  --cache-dir ~/.local/share/epochnbd/my-vault-cache \
  --mount-dir ~/vault \
  --size 2G

• Opens a Google sign-in flow for this session (or prints a URL + verification code prompt automatically over SSH; pass --no-browser anytime to force copy-paste mode).
• Initializes manifest.json on GCS, attaches/dev/nbd0 , formats LUKS2 (type uppercaseYES + passphrase), formatsext4 , mounts~/vault owned by your user, and checkpoints Epoch 1.
• (If init is interrupted during interactive LUKS formatting before writing a header, run ./epochnbd-vault.sh resume-init ... with the same flags to finish formatting safely.)

./epochnbd-vault.sh down \
  --bin ./bin/epochnbd \
  --cache-dir ~/.local/share/epochnbd/my-vault-cache \
  --mount-dir ~/vault

• Syncs and unmounts ~/vault , closes/dev/mapper/epochnbd_vault , stops the NBD daemon (up all remaining dirty chunks and releasing the GCS lease), and revokes + wipes the RAM session token.

./epochnbd-vault.sh up \
  --bin ./bin/epochnbd \
  --backend gs://my-bucket/my-vault \
  --cache-dir ~/.local/share/epochnbd/my-vault-cache \
  --mount-dir ~/vault

If your previous machine lost power before running down (leaving a foreign lease_holder in manifest.json), start with an empty --cache-dir on the new host:

./epochnbd-vault.sh recover \
  --bin ./bin/epochnbd \
  --backend gs://my-bucket/my-vault \
  --cache-dir ~/.local/share/epochnbd/my-vault-cache \
  --mount-dir ~/vault

Omit --mount-dir whenever you want direct control over /dev/nbd0 (custom encryption, different filesystems, partitions, or unencrypted volumes):

./epochnbd-vault.sh init \
  --bin ./bin/epochnbd \
  --backend gs://my-bucket/raw-disk \
  --cache-dir ~/.local/share/epochnbd/raw-cache \
  --size 4G

sudo mkfs.ext4 -F -b 4096 -O ^has_journal,sparse_super2 -E lazy_itable_init=1,nodiscard -G 64 -m 0 /dev/nbd0
sudo mount -o noatime,nodiratime,noinit_itable /dev/nbd0 /mnt/mydisk


sudo umount /mnt/mydisk
./epochnbd-vault.sh down \
  --bin ./bin/epochnbd \
  --cache-dir ~/.local/share/epochnbd/raw-cache

Safety Check in Raw Mode: down inspects lsblk and /sys/block/nbdX/holders before stopping the daemon. If /dev/nbd0 still has an active mount or an open mapper/encryption container, down refuses to detach until you unmount/close it, then runs blockdev --flushbufs before up the final epoch.

Both GCSBackend and GDriveBackend share a unified 4-tier Zero-Disk + ADC credential resolver (ResolveGoogleClientOptions):

1. Direct In-Memory Access Token (EPOCHNBD_GOOGLE_TOKEN) Pass a raw OAuth2 access token in EPOCHNBD_GOOGLE_TOKEN (e.g. from Google Colabgoogle.colab.auth orgcloud auth print-access-token ).epochnbd constructs an in-memoryStaticTokenSource and immediately unsetsEPOCHNBD_GOOGLE_TOKEN from the process environment—zero files are read or written to disk.
2. In-Memory Credential JSON (EPOCHNBD_GOOGLE_CREDENTIALS_JSON) Pass authorized-user or service-account JSON directly viaEPOCHNBD_GOOGLE_CREDENTIALS_JSON .epochnbd parses the credentials in memory, zeroes the raw byte buffer, and unsets the environment variable immediately.
3. Ephemeral RAM Credential File Auto-Unlinking (GOOGLE_APPLICATION_CREDENTIALS) - Per-session browser login ( gs:// & gdrive:// default via epochnbd-vault.sh): Creates an isolated 0700 directory in RAM (/run/user/$UID/epochnbd/... or/dev/shm/... , never touching~/.config/gcloud ) and runsgcloud auth application-default login (forgs:// ) orgcloud auth login --enable-gdrive-access --update-adc (forgdrive:// ). Pass--no-browser over Remote Desktop or SSH to use Google's verification-code page instead of alocalhost redirect.

• Custom OAuth 2.0 Desktop Client (--oauth-client for gcloud-less hosts): Runs a built-in OAuth2 + PKCE flow (local loopback or SSH copy-paste) requesting GCS or Drive scopes (drive.file +drive ).
• Whenever GOOGLE_APPLICATION_CREDENTIALS points into/dev/shm/ ,/run/user/ , orEPOCHNBD_UNLINK_CREDS=1 is set,epochnbd loads the refresh token into Go heap RAM, overwrites the file with zeroes, andunlink s it immediately on startup.

4. Standard Disk ADC Fallback (with SUDO_USER Support) Falls back to/home/$SUDO_USER/.config/gcloud/application_default_credentials.json when invoked undersudo , or standardgoogle.FindDefaultCredentials .

You can also run the Go binary directly without the helper script (for example, with file:///path local backends or TCP NBD listeners):

Usage:
  epochnbd <subcommand> [flags]

Subcommands:
  init    Initialize a new volume manifest on the cloud/local backend
  mount   Mount the cloud-backed block device (/dev/nbdX and/or TCP NBD server)
  sync    Trigger an immediate offline checkpoint and compaction on a volume
  auth    Authorize a per-session GCS OAuth2 token via browser (PKCE)
  revoke  Revoke and zero-wipe an ephemeral per-session OAuth2 credential file

Flag	Default	Description
--backend	(required)	gs://bucket/prefix ,gdrive://<folder_id_or_"root">/subpath , orfile:///abs/path
--cache-dir	(required)	Local directory for cache.img ,wal.log , andstate.meta
--device	""	Kernel NBD device to attach (e.g. /dev/nbd0 , requires root/CAP_SYS_ADMIN )
--listen	""	Serve NBD over TCP or Unix socket (e.g. 127.0.0.1:10809 orunix:///tmp/nbd.sock )
--sync-mode	local-wal	local-wal (fast local WAL fsync) orcloud-strict (upload WAL segment on every flush)
--compaction-interval	5s	Idle debounce duration before dirty chunks compact into a new cloud epoch
--max-wal-size	33554432	Maximum wal.log bytes (32 MiB ) before forcing immediate compaction
--max-cache-size	128M	Maximum local cache.img size before LRU hole punching deallocates clean blocks (0 to disable)
--max-dirty-size	64M	Maximum uncompacted dirty chunk bytes before applying write backpressure ( 0 to disable)
--workers	4	Concurrent chunk upload workers during epoch compaction
--force-unlock	false	Force-reclaim a stale lease_holder from a crashed foreign host

Send SIGUSR1 to a running mount process at any time to force an immediate cloud epoch upload without unmounting:

sudo kill -USR1 "$(sudo cat /path/to/cache/epochnbd.pid)"

epochnbd enforces conservative default bounds on local cache and uncompacted dirty data:

Option	Default	Purpose
--max-cache-size	128M	Physical host disk cap for cache.img via LRU kernel hole punching (0 to disable)
--max-dirty-size	64M	Dirty uncompacted chunk threshold before applying writer backpressure ( 0 to disable)

1. Prevent Host Disk Exhaustion on Massive Cloud Volumes: A virtual disk can be 512 GiB, 2 TiB, or larger. When running on virtual machines, dev containers, or laptops with limited host disk space (e.g. 5–15 GB free), copying large file trees or runningrsync with unbounded caching would quickly fill the entire host root filesystem, leading to system lockups or forced reboots.
2. Bridge the Disk I/O vs. Cloud Upload Speed Gap: Local applications can write data to Linux block devices at hundreds of megabytes or gigabytes per second, while cloud WAN upload bandwidth (especially under Google Drive rate limits) is typically much lower. Without dirty backpressure (--max-dirty-size 64M ), local writes outpace cloud compaction uploads, inflatingwal.log and dirty cache indefinitely. Bounding dirty data to 64 MiB throttles writes cleanly to cloud upload throughput.
3. Safe by Default Anywhere: The 128M/64M defaults ensureepochnbd can safely run in low-memory/low-disk environments (like VPS or CI containers) right out of the box.

Cache limits are client-side operational policies, completely independent of the remote volume's authoritative cloud manifest (manifest.json). You can change --max-cache-size and --max-dirty-size freely between sessions whenever running up or mount:

• Decreasing --max-cache-size (e.g. from 1G to 128M): You donot need to delete cache files or run recovery. On startup, the engine scans the existing cache, detects that cached blocks exceed the new lower limit, and automatically executes kernel hole punching (fallocate(FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE) ) on the oldest clean chunks, immediately returning physical disk blocks to the host OS.
• Increasing --max-cache-size (e.g. from 128M to 2G): Existing cached data remains intact. The cache simply accommodates more clean chunks before LRU eviction triggers.
• Disabling Limits: Pass--max-cache-size 0 and--max-dirty-size 0 if you have plenty of local disk and want unlimited local caching.

Warning

Crash Recovery, Teardown Latency, and the Risks of Unbounded Dirty Cache:

Teardown & Unmount Latency: When detaching the volume or shutting down cleanly (epochnbd-vault.sh down ), the daemon flushes all uncompacted dirty chunks into a final cloud epoch before disconnecting. If--max-dirty-size is set excessively large (or0 for unbounded dirty accumulation), a heavy write burst right before detachment leaves hundreds of megabytes or gigabytes of dirty chunks that must all be uploaded across the network before unmounting and NBD detachment can cleanly complete. 2. Crash Durability vs. Local Cache Loss: By design, the data volume will survive sudden machine crashes, power failure, or unclean detaches (kill -9 ) without data loss—recovering fully upon the next restart—as long as the local cache files (cache.img, wal.log, state.meta) remain intact on the host . However, in the accident that the host machine fails permanently and its local cache is lost before dirty data is compacted to the cloud, any uncompacted dirty writes will be lost (the volume rolls back to the last committed cloud epoch). 3. Post-Crash Re-attachment Latency: Even when local cache files are completely intact after a crash, Pre-Mount Recovery must flush and upload all leftover uncompacted dirty chunks to the cloudbefore the NBD block device is re-exposed to the OS. If--max-dirty-size was set very large, re-attaching the volume after an unclean shutdown can take a very long time while those gigabytes of dirty chunks are uploaded.

Recommendation: Keeping --max-dirty-size bounded (e.g. 64M– 512M) ensures clean detachment finishes quickly, bounds post-crash re-attachment recovery time, and minimizes dirty data exposed to local drive failure.

Scheme	Example URI	Atomic CAS Mechanism	Status
gs://	gs://my-bucket/subdir	GCS Object Generation ( ifGenerationMatch )	Production
gdrive://	gdrive://root/sparrow-vault orgdrive://<folderId>/vault	Mutex + live fileId:md5Checksum verification (trashed = false )	Production
file://	file:///var/lib/epochnbd-store	flock + SHA-256 verification +fsync + atomicrename	Production
s3://	s3://my-bucket/subdir	If-Match /If-None-Match conditional writes	Interface stub (roadmap)

All unit, race-condition, fault-injection, cold-cache read-modify-write, and mock-GCS/OAuth tests run strictly inside temporary directories (t.TempDir()) without requiring root privileges or cloud credentials:

CGO_ENABLED=1 go test -v -race ./...

source & further reading

github.com — original article Turn Superlogical's Rex into an AI Harness Nouli – A decision model TUI to branch and blend multiple histories Show HN: Jik – a self-hosted language using region-based memory management