{"slug": "show-hn-epochnbd-crash-consistent-linux-nbd-block-device-on-gcs-google-drive", "title": "Show HN: Epochnbd – Crash-Consistent Linux NBD Block Device on GCS/Google Drive", "summary": "A developer released epochnbd, an experimental pure-Go Linux Network Block Device engine that exposes a raw block device backed by Google Cloud Storage, Google Drive, or a local directory, with a CRC32-verified write-ahead log and atomic compare-and-swap epoch checkpoints. The project, built with Gemini Argon and published as a Show HN post, uses 512 KiB chunks on GCS and local storage and 4 MiB chunks on Google Drive, and advances its manifest.json only after every dirty chunk in an epoch uploads successfully via GCS ifGenerationMatch preconditions. The author labels epochnbd a personal hobby project provided \"AS IS\" without warranty, warning of data loss, corruption, and cloud billing risks and urging independent backups.", "body_md": "Caution\n\n**Disclaimer — Experimental Hobby Project (Use at Your Own Risk)**\n\n`epochnbd` is a personal hobby project developed with **Gemini Argon**. It is provided **\"AS IS\"** without warranty of any kind. Storage engines and kernel block devices carry inherent risks of bugs, corruption, or cloud billing surprises. **Use this software entirely at your own risk**—always maintain independent backups of important data. The author assumes **no responsibility or liability** for any data loss, corruption, system instability, or other damages resulting from its use.\n\n**`epochnbd`** is a pure-Go (` CGO_ENABLED=0`) Linux Network Block Device (NBD) engine that exposes a true POSIX raw block device (`/dev/nbdX` or TCP NBD export) backed by **Google Cloud Storage (`gs://`)**, **Google Drive (` gdrive://`)**, or a **local directory (` file://`)**, with local sparse caching, a CRC32-verified Write-Ahead Log (WAL), and atomic Compare-And-Swap (CAS) epoch checkpoints.\n\nUnlike FUSE object-storage mounts (`gcsfuse`, `s3fs`), `epochnbd` operates strictly at the **raw block layer**:\n\n- **Full POSIX & Database Compatibility:** Supports`mmap` ,`flock` /`fcntl` locks,`fsync` flush barriers, SQLite in WAL mode, Git repositories, and standard Linux filesystems (`ext4` ,`xfs` ,`btrfs` ).\n- **Bring Your Own Encryption (Zero-Knowledge Cloud Storage):** Stack standard Linux**LUKS2 (`dm-crypt`)** , VeraCrypt, or any block/filesystem encryption on top of`/dev/nbd0` —or run unencrypted when desired. When encrypted at the block layer, cloud storage only ever sees opaque fixed-size ciphertext chunks (`512 KiB` default on GCS/LocalFS,`4 MiB` default on Google Drive).\n- **Zero Torn Cloud Checkpoints:** Dirty chunks are uploaded to immutable, epoch-tagged objects (`chunks/000042_e7.bin` ), and`manifest.json` advances**atomically** via GCS generation preconditions (`ifGenerationMatch` ) only after every dirty chunk in the epoch succeeds.\n\n```\n  Application / SQLite / Git\n              │\n              ▼\n   Filesystem (ext4 / xfs / btrfs)        ┐\n              │                           │ Optional user-space / kernel stack\n   Encryption (LUKS2 / VeraCrypt / none)  ┘ (Managed automatically if --mount-dir is set)\n              │\n              ▼\n      Linux Kernel /dev/nbd0\n              │  (NBD_CMD_READ / WRITE / FLUSH / DISC)\n              ▼\n┌─────────────────────────────────────────────────────────────┐\n│                     epochnbd Daemon                         │\n│                                                             │\n│  ┌─────────────────────┐      ┌──────────────────────────┐  │\n│  │  Sparse cache.img   │◄────►│  CRC32 Write-Ahead Log   │  │\n│  │ + Presence/Dirty    │      │  (wal.log + fsync)       │  │\n│  │   Chunk Bitmaps     │      └──────────────────────────┘  │\n│  └──────────┬──────────┘                   │                │\n│             │ Debounced Compactor          │ (cloud-strict) │\n│             ▼                              ▼                │\n│  ┌───────────────────────────────────────────────────────┐  │\n│  │ StorageBackend: GCS / Google Drive / LocalFS          │  │\n│  └───────────────────────────────────────────────────────┘  │\n└─────────────────────────────┬───────────────────────────────┘\n                              │\n                              ▼\n     Cloud Bucket / Drive Folder (e.g. gs://my-bucket/vault or gdrive://root/vault)\n         ├── manifest.json             (Atomic CAS pointer + single-writer lease)\n         ├── chunks/000000_e1.bin      (Immutable 512 KiB blocks per epoch)\n         ├── chunks/000003_e2.bin\n         └── wal/seq_00000012.seg      (Optional cloud-strict WAL segments)\n<cache-dir>/\n├── cache.img            # Sparse raw disk image (apparent size = --size; uses disk only for touched blocks)\n├── wal.log              # Append-only binary WAL with per-record IEEE CRC32 checksums\n├── state.meta           # Atomic metadata (0600): volume_uuid, local_epoch, present_bits, dirty_bits\n├── epochnbd.pid         # Active daemon PID while mounted (managed by epochnbd-vault.sh, removed on down)\n└── epochnbd.daemon.log  # Daemon startup, recovery, and epoch compaction logs (managed by epochnbd-vault.sh)\n```\n\n- **`cache.img` (Sparse Block Cache):** Created via`Truncate(DiskSize)` as a Linux sparse file. Untouched regions take`0` bytes on your local SSD. Chunks are faulted in from cloud storage on first read (or partial-chunk read-modify-write) and served locally thereafter.\n- **`wal.log` (Write-Ahead Log):** Stores sequential`Write` and`FlushBarrier` records (`\"CNBDWAL1\"` header) before data is compacted to cloud storage. Trimmed automatically once an epoch commits to`manifest.json` ; torn tail records from power loss are automatically truncated to the last valid CRC32 boundary on startup.\n- **`state.meta` (Persistent Cache State):** Atomically written (`tmp` +`fsync` +`rename` ) JSON tracking`volume_uuid` ,`local_epoch` , cached per-chunk epochs,`PresentBitmap` , and`DirtyBitmap` :\n  - Prevents accidentally attaching a cache directory belonging to a different volume UUID.\n  - Enables **warm-cache reuse** across`down` /`up` cycles: if another machine updated the volume in the cloud,`OpenEngine` compares`state.meta` against`manifest.json` and invalidates*only* the specific chunks whose epoch changed.\n- **When is it safe to delete `<cache-dir>`?**  - **Keep `<cache-dir>`** across normal` down` /`up` sessions for fast warm-cache reads.\n  - **Safe to delete** anytime*after* a clean`down` (all dirty chunks are already committed to the cloud; the next`up` simply starts with a cold sparse cache and faults chunks in on demand).\n  - **Do NOT delete** after an unclean crash (`kill -9` or power loss)*before* running`up` on that machine—`wal.log` and`cache.img` hold any uncompacted local writes that Pre-Mount Recovery will replay and push to the cloud on next start.\n\n| Object Key | Purpose | \n|---|---|\n| `manifest.json` | Plain JSON volume descriptor: `volume_uuid` ,`disk_size` ,`chunk_size` , monotonic`epoch` ,`last_wal_seq` ,`chunk_epochs` map (`chunkIdx -> epoch` ), and`lease_holder` . Updated**only** via atomic CAS. | \n| `chunks/%06d_e%d.bin` | Immutable `ChunkSize` (default`512 KiB` for`gs://` /`file://` ,`4 MiB` for`gdrive://` ) raw block payload for chunk`idx` written in epoch`e` . Unmapped chunk indices are sparse (`0x00` ). Superseded chunk epochs are garbage-collected after`manifest.json` advances. | \n| `wal/seq_%08d.seg` | *(Only in `--sync-mode=cloud-strict`)* Remote WAL delta segments uploaded on every` fsync` barrier before epoch compaction folds and deletes them. | \n\nWhen using `epochnbd-vault.sh` with a `gs://` backend (without a persistent `GOOGLE_APPLICATION_CREDENTIALS` key), session OAuth credentials live **only in RAM** at `/run/user/$UID/epochnbd/gcloud-nbd0` (or `/dev/shm/epochnbd-$UID/...`) and are revoked + deleted automatically during `down`.\n\n1. \n**Atomic Epoch Switching & Selective GC:** Writes hit local sparse`cache.img` +`wal.log` immediately. In the background (every`5s` of write idle, when`wal.log >= 32 MiB` , on`SIGUSR1` , or on graceful shutdown), the compactor snapshots dirty chunks, uploads them in parallel as`chunks/<idx>_e<Epoch+1>.bin` , commits`manifest.json` via atomic CAS, trims compacted WAL records, and garbage-collects only superseded chunk versions.\n2. \n**Two Selectable Flush Barrier Modes (`--sync-mode`):**\n  - `local-wal`*(default)* :` fsync` (`NBD_CMD_FLUSH` ) persists to local`wal.log` +`cache.img` for fast local latency while compacting epochs asynchronously to the cloud.\n  - `cloud-strict` : Every`fsync` barrier also uploads a WAL delta segment (`wal/seq_<N>.seg` ) to cloud storage before acknowledging the flush to the kernel (`RPO = 0` even if the local machine is destroyed mid-session).\n **Durability Summary (What happens on a crash?):**\n  - **Process crash or power loss (`up` on the same machine):****Zero data loss** in both`local-wal` and`cloud-strict` modes for all`fsync` ed writes, because Pre-Mount Recovery replays local`wal.log` and pushes a new cloud epoch before mounting.\n  - **Total machine/SSD destruction (`recover` on a new machine):** Rolls back cleanly to the last committed cloud epoch (~5s of write idle in`local-wal` mode), or**zero data loss (`RPO = 0`)** in` cloud-strict` mode.\n3. \n**Pre-Mount Crash Recovery:** Before opening the NBD listener or attaching`/dev/nbdX` ,`OpenEngine` checks`state.meta` ,`wal.log` , and remote`wal/seq_*.seg` blobs. Any uncommitted writes from a prior crash (`kill -9` or power loss) are replayed, compacted into a new immutable epoch, and committed via CAS**before** the block device becomes visible to the OS.\n4. \n**Single-Writer Lease & Split-Brain Fencing:**`manifest.json` records`lease_holder` (`hostname:pid:instance_uuid` ). Same-host restarts automatically reclaim stale crash leases; foreign hosts are blocked unless`--force-unlock` is explicitly passed (`recover` ). If another writer steals the lease mid-session, GCS HTTP`412 Precondition Failed` immediately fences the stale node from overwriting`manifest.json` .\n5. \n**Triple Pre-Flight `init` Protection:**`init` refuses to write a single byte if:\n  - `manifest.json` already exists at the backend,\n  - *any* objects exist under the backend prefix (`chunks/` ,`wal/` , etc., even if`manifest.json` was accidentally deleted), or\n  - `--cache-dir` contains leftover state (`cache.img` ,`wal.log` ,`state.meta` ) from an existing volume.\n6. \n**Colab-Style Per-Session Browser Auth (Zero Secret Files):**`epochnbd-vault.sh` authenticates GCS per session via your browser (works both on a** local desktop** and over**remote SSH** copy-paste), stores the temporary token strictly in**RAM (`tmpfs`)** , and automatically**revokes + zero-wipes** the token after flushing the final epoch on`down` .\n\n- **OS:** Linux with the`nbd` kernel module (`sudo modprobe nbd` )\n- **Build:** Go`1.22+` (builds a static`CGO_ENABLED=0` binary)\n- **System Utilities:**`util-linux` (`blockdev` ,`lsblk` ,`mount` ,`umount` ,`findmnt` )\n- **Optional (for Managed LUKS2+ext4 Vault Mode):**`cryptsetup` (LUKS2) and`e2fsprogs` (`mkfs.ext4` ,`e2fsck` )\n- **Optional (for zero-secret-file GCS browser login):**[Google Cloud CLI (`gcloud`)](https://cloud.google.com/sdk/docs/install)\n\n```\nCGO_ENABLED=0 go build -o ./bin/epochnbd ./cmd/epochnbd\nchmod +x ./epochnbd-vault.sh\n```\n\n`epochnbd-vault.sh` supports two workflows depending on whether `--mount-dir` is passed:\n\n- **Managed Vault Mode (`--mount-dir <dir>`):** Automates LUKS2 (` aes-xts-plain64` , 512-bit key, Argon2id, 4K sectors) + sparse-optimized`ext4` (`-O ^has_journal,sparse_super2 -E lazy_itable_init=1,nodiscard -G 64 -m 0` ) + mount (`-o noatime,nodiratime,noinit_itable` ) so both small (`256 MiB` ) and multi-terabyte (`2 TiB+` ) encrypted volumes initialize in seconds without pre-zeroing inode tables into LUKS ciphertext chunks.\n- **Raw NBD Mode (omit `--mount-dir`):** Attaches/detaches raw`/dev/nbd0` only, leaving formatting, custom encryption (e.g. VeraCrypt, ZFS, custom LUKS), or unencrypted filesystems completely up to you.\n\n```\n./epochnbd-vault.sh init \\\n  --bin ./bin/epochnbd \\\n  --backend gs://my-bucket/my-vault \\\n  --cache-dir ~/.local/share/epochnbd/my-vault-cache \\\n  --mount-dir ~/vault \\\n  --size 2G\n```\n\n- Opens a Google sign-in flow for this session (or prints a URL + verification code prompt automatically over SSH; pass `--no-browser` anytime to force copy-paste mode).\n- Initializes `manifest.json` on GCS, attaches`/dev/nbd0` , formats LUKS2 (type uppercase`YES` + passphrase), formats`ext4` , mounts`~/vault` owned by your user, and checkpoints Epoch 1.\n- *(If `init` is interrupted during interactive LUKS formatting before writing a header, run `./epochnbd-vault.sh resume-init ...` with the same flags to finish formatting safely.)*\n\n```\n./epochnbd-vault.sh down \\\n  --bin ./bin/epochnbd \\\n  --cache-dir ~/.local/share/epochnbd/my-vault-cache \\\n  --mount-dir ~/vault\n```\n\n- Syncs and unmounts `~/vault` , closes`/dev/mapper/epochnbd_vault` , stops the NBD daemon (uploading all remaining dirty chunks and releasing the GCS lease), and revokes + wipes the RAM session token.\n\n```\n./epochnbd-vault.sh up \\\n  --bin ./bin/epochnbd \\\n  --backend gs://my-bucket/my-vault \\\n  --cache-dir ~/.local/share/epochnbd/my-vault-cache \\\n  --mount-dir ~/vault\n```\n\nIf your previous machine lost power before running `down` (leaving a foreign `lease_holder` in `manifest.json`), start with an **empty** `--cache-dir` on the new host:\n\n```\n./epochnbd-vault.sh recover \\\n  --bin ./bin/epochnbd \\\n  --backend gs://my-bucket/my-vault \\\n  --cache-dir ~/.local/share/epochnbd/my-vault-cache \\\n  --mount-dir ~/vault\n```\n\nOmit `--mount-dir` whenever you want direct control over `/dev/nbd0` (custom encryption, different filesystems, partitions, or unencrypted volumes):\n\n```\n# 1. Initialize and attach raw /dev/nbd0\n./epochnbd-vault.sh init \\\n  --bin ./bin/epochnbd \\\n  --backend gs://my-bucket/raw-disk \\\n  --cache-dir ~/.local/share/epochnbd/raw-cache \\\n  --size 4G\n\n# 2. Use /dev/nbd0 however you like (e.g., sparse-optimized ext4, XFS, Btrfs, VeraCrypt, etc.)\nsudo mkfs.ext4 -F -b 4096 -O ^has_journal,sparse_super2 -E lazy_itable_init=1,nodiscard -G 64 -m 0 /dev/nbd0\nsudo mount -o noatime,nodiratime,noinit_itable /dev/nbd0 /mnt/mydisk\n\n# ... read & write files in /mnt/mydisk ...\n\n# 3. Unmount/close your upper layers first, then detach & sync to cloud\nsudo umount /mnt/mydisk\n./epochnbd-vault.sh down \\\n  --bin ./bin/epochnbd \\\n  --cache-dir ~/.local/share/epochnbd/raw-cache\n```\n\n**Safety Check in Raw Mode:** `down` inspects `lsblk` and `/sys/block/nbdX/holders` before stopping the daemon. If `/dev/nbd0` still has an active mount or an open mapper/encryption container, `down` refuses to detach until you unmount/close it, then runs `blockdev --flushbufs` before uploading the final epoch.\n\nBoth `GCSBackend` and `GDriveBackend` share a unified 4-tier Zero-Disk + ADC credential resolver (`ResolveGoogleClientOptions`):\n\n1. **Direct In-Memory Access Token (`EPOCHNBD_GOOGLE_TOKEN`)** Pass a raw OAuth2 access token in` EPOCHNBD_GOOGLE_TOKEN` (e.g. from Google Colab`google.colab.auth` or`gcloud auth print-access-token` ).`epochnbd` constructs an in-memory`StaticTokenSource` and immediately unsets`EPOCHNBD_GOOGLE_TOKEN` from the process environment—zero files are read or written to disk.\n2. **In-Memory Credential JSON (`EPOCHNBD_GOOGLE_CREDENTIALS_JSON`)** Pass authorized-user or service-account JSON directly via`EPOCHNBD_GOOGLE_CREDENTIALS_JSON` .`epochnbd` parses the credentials in memory, zeroes the raw byte buffer, and unsets the environment variable immediately.\n3. **Ephemeral RAM Credential File Auto-Unlinking (`GOOGLE_APPLICATION_CREDENTIALS`)**  - **Per-session browser login (` gs://` & `gdrive://` default via `epochnbd-vault.sh`):** Creates an isolated` 0700` directory in RAM (`/run/user/$UID/epochnbd/...` or`/dev/shm/...` , never touching`~/.config/gcloud` ) and runs`gcloud auth application-default login` (for`gs://` ) or`gcloud auth login --enable-gdrive-access --update-adc` (for`gdrive://` ). Pass`--no-browser` over Remote Desktop or SSH to use Google's verification-code page instead of a`localhost` redirect.\n  - **Custom OAuth 2.0 Desktop Client (`--oauth-client` for `gcloud`-less hosts):** Runs a built-in OAuth2 + PKCE flow (local loopback or SSH copy-paste) requesting GCS or Drive scopes (`drive.file` +`drive` ).\n  - Whenever `GOOGLE_APPLICATION_CREDENTIALS` points into`/dev/shm/` ,`/run/user/` , or`EPOCHNBD_UNLINK_CREDS=1` is set,`epochnbd` loads the refresh token into Go heap RAM, overwrites the file with zeroes, and`unlink` s it immediately on startup.\n4. **Standard Disk ADC Fallback (with `SUDO_USER` Support)** Falls back to`/home/$SUDO_USER/.config/gcloud/application_default_credentials.json` when invoked under`sudo` , or standard`google.FindDefaultCredentials` .\n\nYou can also run the Go binary directly without the helper script (for example, with `file:///path` local backends or TCP NBD listeners):\n\n```\nUsage:\n  epochnbd <subcommand> [flags]\n\nSubcommands:\n  init    Initialize a new volume manifest on the cloud/local backend\n  mount   Mount the cloud-backed block device (/dev/nbdX and/or TCP NBD server)\n  sync    Trigger an immediate offline checkpoint and compaction on a volume\n  auth    Authorize a per-session GCS OAuth2 token via browser (PKCE)\n  revoke  Revoke and zero-wipe an ephemeral per-session OAuth2 credential file\n```\n\n| Flag | Default | Description | \n|---|---|---|\n| `--backend` | *(required)* | `gs://bucket/prefix` ,`gdrive://<folder_id_or_\"root\">/subpath` , or`file:///abs/path` | \n| `--cache-dir` | *(required)* | Local directory for `cache.img` ,`wal.log` , and`state.meta` | \n| `--device` | `\"\"` | Kernel NBD device to attach (e.g. `/dev/nbd0` , requires root/`CAP_SYS_ADMIN` ) | \n| `--listen` | `\"\"` | Serve NBD over TCP or Unix socket (e.g. `127.0.0.1:10809` or`unix:///tmp/nbd.sock` ) | \n| `--sync-mode` | `local-wal` | `local-wal` (fast local WAL fsync) or`cloud-strict` (upload WAL segment on every flush) | \n| `--compaction-interval` | `5s` | Idle debounce duration before dirty chunks compact into a new cloud epoch | \n| `--max-wal-size` | `33554432` | Maximum `wal.log` bytes (`32 MiB` ) before forcing immediate compaction | \n| `--max-cache-size` | `128M` | Maximum local `cache.img` size before LRU hole punching deallocates clean blocks (`0` to disable) | \n| `--max-dirty-size` | `64M` | Maximum uncompacted dirty chunk bytes before applying write backpressure ( `0` to disable) | \n| `--workers` | `4` | Concurrent chunk upload workers during epoch compaction | \n| `--force-unlock` | `false` | Force-reclaim a stale `lease_holder` from a crashed foreign host | \n\nSend `SIGUSR1` to a running `mount` process at any time to force an immediate cloud epoch upload without unmounting:\n\n```\nsudo kill -USR1 \"$(sudo cat /path/to/cache/epochnbd.pid)\"\n```\n\n`epochnbd` enforces conservative default bounds on local cache and uncompacted dirty data:\n\n| Option | Default | Purpose | \n|---|---|---|\n| `--max-cache-size` | `128M` | Physical host disk cap for `cache.img` via LRU kernel hole punching (`0` to disable) | \n| `--max-dirty-size` | `64M` | Dirty uncompacted chunk threshold before applying writer backpressure ( `0` to disable) | \n\n1. **Prevent Host Disk Exhaustion on Massive Cloud Volumes:** A virtual disk can be 512 GiB, 2 TiB, or larger. When running on virtual machines, dev containers, or laptops with limited host disk space (e.g. 5–15 GB free), copying large file trees or running`rsync` with unbounded caching would quickly fill the entire host root filesystem, leading to system lockups or forced reboots.\n2. **Bridge the Disk I/O vs. Cloud Upload Speed Gap:** Local applications can write data to Linux block devices at hundreds of megabytes or gigabytes per second, while cloud WAN upload bandwidth (especially under Google Drive rate limits) is typically much lower. Without dirty backpressure (`--max-dirty-size 64M` ), local writes outpace cloud compaction uploads, inflating`wal.log` and dirty cache indefinitely. Bounding dirty data to 64 MiB throttles writes cleanly to cloud upload throughput.\n3. **Safe by Default Anywhere:** The 128M/64M defaults ensure`epochnbd` can safely run in low-memory/low-disk environments (like VPS or CI containers) right out of the box.\n\nCache limits are **client-side operational policies**, completely independent of the remote volume's authoritative cloud manifest (`manifest.json`). You can change `--max-cache-size` and `--max-dirty-size` freely between sessions whenever running `up` or `mount`:\n\n- **Decreasing `--max-cache-size` (e.g. from `1G` to `128M`):** You do**not** need to delete cache files or run recovery. On startup, the engine scans the existing cache, detects that cached blocks exceed the new lower limit, and automatically executes kernel hole punching (`fallocate(FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE)` ) on the oldest clean chunks, immediately returning physical disk blocks to the host OS.\n- **Increasing `--max-cache-size` (e.g. from `128M` to `2G`):** Existing cached data remains intact. The cache simply accommodates more clean chunks before LRU eviction triggers.\n- **Disabling Limits:** Pass`--max-cache-size 0` and`--max-dirty-size 0` if you have plenty of local disk and want unlimited local caching.\n\nWarning\n\n**Crash Recovery, Teardown Latency, and the Risks of Unbounded Dirty Cache:**\n\n1. \n**Teardown & Unmount Latency:** When detaching the volume or shutting down cleanly (`epochnbd-vault.sh down` ), the daemon flushes all uncompacted dirty chunks into a final cloud epoch before disconnecting. If`--max-dirty-size` is set excessively large (or`0` for unbounded dirty accumulation), a heavy write burst right before detachment leaves hundreds of megabytes or gigabytes of dirty chunks that must all be uploaded across the network before unmounting and NBD detachment can cleanly complete.\n2. \n**Crash Durability vs. Local Cache Loss:** By design, the data volume will survive sudden machine crashes, power failure, or unclean detaches (`kill -9` ) without data loss—recovering fully upon the next restart—**as long as the local cache files (`cache.img`, `wal.log`, `state.meta`) remain intact on the host** . However, in the accident that the host machine fails permanently and its local cache is lost before dirty data is compacted to the cloud, any uncompacted dirty writes will be lost (the volume rolls back to the last committed cloud epoch).\n3. \n**Post-Crash Re-attachment Latency:** Even when local cache files are completely intact after a crash, Pre-Mount Recovery must flush and upload all leftover uncompacted dirty chunks to the cloud*before* the NBD block device is re-exposed to the OS. If`--max-dirty-size` was set very large, re-attaching the volume after an unclean shutdown can take a very long time while those gigabytes of dirty chunks are uploaded.\n\n**Recommendation:** Keeping `--max-dirty-size` bounded (e.g. `64M`–` 512M`) ensures clean detachment finishes quickly, bounds post-crash re-attachment recovery time, and minimizes dirty data exposed to local drive failure.\n\n| Scheme | Example URI | Atomic CAS Mechanism | Status | \n|---|---|---|---|\n| `gs://` | `gs://my-bucket/subdir` | GCS Object Generation ( `ifGenerationMatch` ) | Production | \n| `gdrive://` | `gdrive://root/sparrow-vault` or`gdrive://<folderId>/vault` | Mutex + live `fileId:md5Checksum` verification (`trashed = false` ) | Production | \n| `file://` | `file:///var/lib/epochnbd-store` | `flock` + SHA-256 verification +`fsync` + atomic`rename` | Production | \n| `s3://` | `s3://my-bucket/subdir` | `If-Match` /`If-None-Match` conditional writes | Interface stub (roadmap) | \n\nAll unit, race-condition, fault-injection, cold-cache read-modify-write, and mock-GCS/OAuth tests run strictly inside temporary directories (`t.TempDir()`) without requiring root privileges or cloud credentials:\n\n```\nCGO_ENABLED=1 go test -v -race ./...\n```\n\n", "url": "https://wpnews.pro/news/show-hn-epochnbd-crash-consistent-linux-nbd-block-device-on-gcs-google-drive", "canonical_source": "https://github.com/c789a123/epochnbd", "published_at": "2026-11-01 20:58:30+00:00", "updated_at": "2026-11-01 21:21:08.955756+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["epochnbd", "Google Cloud Storage", "Google Drive", "Gemini Argon", "Linux"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-epochnbd-crash-consistent-linux-nbd-block-device-on-gcs-google-drive", "markdown": "https://wpnews.pro/news/show-hn-epochnbd-crash-consistent-linux-nbd-block-device-on-gcs-google-drive.md", "text": "https://wpnews.pro/news/show-hn-epochnbd-crash-consistent-linux-nbd-block-device-on-gcs-google-drive.txt", "jsonld": "https://wpnews.pro/news/show-hn-epochnbd-crash-consistent-linux-nbd-block-device-on-gcs-google-drive.jsonld"}}