{"slug": "running-blackwell-rtx-5090-headless-24-7-the-real-boss-is-the-gsp-hang-and-i-it", "title": "Running Blackwell (RTX 5090) Headless 24/7: The Real Boss Is the GSP Hang, and I Took It With a \"Assume It Breaks\" Watchdog", "summary": "A developer running an RTX 5090 (Blackwell) headless on NixOS for 24/7 local LLM inference documented the GSP heartbeat timeout hang, an unresolved bug tracked in open-gpu-kernel-modules issues #1080 and #1111, and built a declarative systemd watchdog that polls nvidia-smi every five minutes and reboots the machine if the GPU stops responding for 60 seconds. Because the open driver ignores the NVreg_EnableGpuFirmware=0 workaround and in-place resets fail, the developer concluded reboot is the only reliable recovery, and also applied a 400W power cap as an anecdotal mitigation.", "body_md": "In the [previous post](https://wada-dev.com/en/blog/nixos-home-ai-server-install-gotchas) I covered picking NixOS for my home AI server and doing the base install. Once the OS is in, the next step is enabling the GPU and getting to the actual goal — running local LLMs 24/7. But this is where the hardest obstacle lives: the **GSP heartbeat timeout**, an as-yet-unresolved hang bug.\n\nThe GPU is an **RTX 5090 (Blackwell generation)**. Repurposing a latest-gen consumer card as a server is **pioneer territory** with almost no community precedent for stable operation. So this time, instead of \"run it and deal with problems later,\" I decided to **scout the landmines before stepping on them**. This post is a write-up of the known issues that scouting surfaced, and the watchdog design I built assuming the hang *will* eventually hit.\n\nTo be honest up front: this isn't a war story of \"my GPU crashed a dozen times.\" It's the story of **scouting ahead and preparing a catch for when it breaks**. The first-hand part I'm writing about is that design decision.\n\nLet's clear the parts you can settle by just looking them up, before they bite.\n\n`nvidia-open`, no choice\nBlackwell is **completely unsupported** by the proprietary driver (per NVIDIA). There's no alternative, and NixOS **enforces** declaring `open` via an assertion for driver >= 560. Headless operation also wants `nvidiaPersistenced` — the right setting to keep the GPU awake even when headless.\n\n```\nhardware.nvidia = {\n  open = true;               # the only option on Blackwell\n  nvidiaPersistenced = true; # keeps the GPU initialized while headless\n};\n```\n\n`cudaPackages` is intentionally held back on the 12.x line, so you have to declare `cudaCapabilities = [\"12.0\"]` explicitly. The official docs' Blackwell example only covers sm_100 (datacenter); consumer sm_120 is a blind spot.`120a`` 120` throws a ptxas error on MXFP4. Ollama is in the same blast radius, so assume you'll verify GPU offload empirically rather than trust it works.\nThat's the \"look it up and the answer is settled\" layer. The next part is the problem.\n\nThis is the scariest thing about 24/7 operation. The GPU's firmware (GSP) suddenly stops responding and takes the whole GPU down with it.\n\n`open-gpu-kernel-modules` has issues #1080 / #1111 open (as of 2026-06). In other words, And the part that really hurts is that **recovery after it happens is hell**.\n\n`open` driver, the classic workaround of `NVreg_EnableGpuFirmware=0` is ignored.\nIn short, \"prevent it with config\" isn't on the table — the reality is \"**if it wedges, your only option is to reboot**.\"\n\nIf you can't prevent it, all you can do is **detect the wedge and recover automatically**. Since an in-place reset doesn't work, the only reliable recovery is a reboot. I wired a declarative systemd watchdog into NixOS:\n\n```\nsystemd = {\n  services.gpu-watchdog = {\n    description = \"Reboot if nvidia-smi stops responding (GSP wedge)\";\n    path = [config.hardware.nvidia.package.bin];\n    script = ''\n      if ! timeout 60 nvidia-smi > /dev/null 2>&1; then\n        echo \"nvidia-smi unresponsive for 60s — rebooting\" \\\n          | systemd-cat -p err -t gpu-watchdog\n        systemctl reboot\n      fi\n    '';\n    serviceConfig.Type = \"oneshot\";\n  };\n\n  timers.gpu-watchdog = {\n    wantedBy = [\"timers.target\"];\n    timerConfig = {\n      OnBootSec = \"5min\";\n      OnUnitActiveSec = \"5min\";\n    };\n  };\n};\n```\n\nEvery 5 minutes it pokes `nvidia-smi`, and **if it doesn't come back within 60 seconds it declares the GPU wedged and runs `systemctl reboot`**. It's blunt, but against an opponent where a reset doesn't work, a reboot is the most reliable move, so I made peace with it. It's unattended, so \"I woke up and it was frozen\" is the only thing I really need to avoid.\n\nI also added a mitigation (anecdote-level, n=1–3): a **power cap**.\n\n```\n# RTX 5090 TDP=575W. Inference and Frigate need well under 400W.\n# Idle stays ~70W regardless. This only caps peak draw.\nscript = ''nvidia-smi -pl 400'';\n```\n\nThere are **PSU-related Xid 119 reports** around the GSP hang (cases resolved by swapping the PSU), so suppressing power spikes is reasonable. This box already runs a native ATX 3.1 1300W Platinum PSU, but I capped peak draw to keep more headroom anyway.\n\n\"Wouldn't Ubuntu, with more reports out there, be more stable?\" is the obvious thought, but my researched answer was No.\n\n**The GSP issue is distro-independent.** It's reported on Mint, Fedora, and others, and Blackwell forces `nvidia-open` on every distro, so the landmine is shared. Switching to Ubuntu buys **more anecdotes but doesn't avoid the crash**. Trading away NixOS's rollback and declarative config for that isn't worth it.\n\nThe landing spot is the same as last time: **stay on NixOS + design assuming a watchdog recovers + keep one Ubuntu Live USB for triage**. Just enough of a safety net to tell \"is this a GPU problem or a Nix-config problem?\" when something jams.\n\nPioneer territory — putting a latest-gen consumer GPU in a server — turns out to have **reset-proof-grade landmines (the GSP heartbeat timeout)** sitting right there. Since you can't prevent it with config, aiming for \"it won't break\" is less realistic than \"**assume it breaks and build automatic recovery**.\"\n\nAnd in moments like this, declarative config quietly pays off. The watchdog and the power cap all live as code in `configuration.nix`, so **the recovery design itself becomes a reproducible asset**. If the SSD dies, I can rebuild the whole catch along with it. It's the same point as the \"peace of mind of rollback\" from last time — the harder the pioneer territory you're crossing, the more this property earns its keep.\n\nWhether I actually hit the moment a GSP wedge happens is something I'll find out as I run it. Once the watchdog's trigger logs start showing up, I'll add an update.\n\n*This post was written with AI assistance and reviewed by the author.*", "url": "https://wpnews.pro/news/running-blackwell-rtx-5090-headless-24-7-the-real-boss-is-the-gsp-hang-and-i-it", "canonical_source": "https://dev.to/kohei_wada/running-blackwell-rtx-5090-headless-247-the-real-boss-is-the-gsp-hang-and-i-took-it-with-a-cjj", "published_at": "2026-10-25 15:36:14+00:00", "updated_at": "2026-10-25 15:43:20.269723+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["NVIDIA", "RTX 5090", "NixOS", "Ollama", "Frigate", "open-gpu-kernel-modules"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-blackwell-rtx-5090-headless-24-7-the-real-boss-is-the-gsp-hang-and-i-it", "markdown": "https://wpnews.pro/news/running-blackwell-rtx-5090-headless-24-7-the-real-boss-is-the-gsp-hang-and-i-it.md", "text": "https://wpnews.pro/news/running-blackwell-rtx-5090-headless-24-7-the-real-boss-is-the-gsp-hang-and-i-it.txt", "jsonld": "https://wpnews.pro/news/running-blackwell-rtx-5090-headless-24-7-the-real-boss-is-the-gsp-hang-and-i-it.jsonld"}}