{"slug": "chasing-driver-truth-and-vram-ghosts", "title": "Chasing Driver Truth and VRAM Ghosts", "summary": "Glad-Labs' Poindexter project shipped fixes on 2026-08-11 to stop silent animation failures by reading VRAM directly from the wan server's /health endpoint via torch.cuda.mem_get_info(0), replacing lagging Prometheus metrics. Release 0.120.0 added schedule_post for MCP and new prime-time slots, and wired up findings.<kind>.cooldown_minutes to activate 37 previously ignored seeded keys.", "body_md": "*What we shipped on 2026-08-11*\n\nWe spent most of today fighting a silent failure in the hero-plate gate, culminating in “size the hero plate from live device VRAM, not a lagging scrape” (PR #3183). We had added a quality floor to prevent generative slop, but that floor became a ceiling that silently disabled motion entirely.\n\nThe first leak was logic: our VRAM check measured raw free memory while `wan`\n\nwas already resident from a previous shot (PR #3181). The system saw 1GB free and skipped the animation, ignoring the fact that the model it needed was already occupying 23GB. We fixed this by introducing `_wan_resident_gb()`\n\n, which pulls `vram_used_mb`\n\nfrom the wan server’s `/health`\n\nendpoint so we can count the resident pool toward the available budget (PR #3181).\n\nThen we hit an ordering problem and a telemetry lag. We were probing VRAM at the top of `_animate_hero`\n\n, but the reclaim logic lived inside `_render_generative_clip`\n\n–meaning we measured the card while `image-gen`\n\nwas still hogging 25GB (PR #3182). Even after flipping the order, Prometheus scrape lag meant our worker was reading data that was ~40s old. We tried riding out the interval by taking the max of four samples (PR #3182), but it wasn’t enough.\n\nThe final fix was to stop trusting the metrics pipeline for real-time gating. By hitting `wan-server /health`\n\nand pulling `device_free_mb`\n\nvia `torch.cuda.mem_get_info(0)`\n\n, we get driver truth in milliseconds rather than Prometheus approximations (PR #3183).\n\nOutside the GPU wars, we cleaned up some technical debt in our testing suite. We found that `tests/unit/cli/test_integrations_youtube_cli.py`\n\nwas using `importlib.reload`\n\non every test, which caused group identity churn and left the CLI operator holding a different group object than the module exposed (PR #3173). We also scrubbed the CI runner docs to remove “fossil” references to `--forked`\n\nand corrected a misleading claim about `.env`\n\nfiles that don’t actually exist in our operator stack (PR #3172).\n\nOn the shipping front, release 0.120.0 landed with `schedule_post`\n\nfor MCP–allowing us to approve and publish slots in one call–and new prime-time slots so night-time publishes can be promoted in the morning (PR #3146). We also finally wired up `findings.<kind>.cooldown_minutes`\n\n(PR #3169), turning 37 seeded keys that were being ignored into actual logic.\n\nWe’re moving away from relying on external telemetry for hardware gates. Driver-level truth is the only way to keep the generative pipeline tight without skipping frames.\n\n*Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.*", "url": "https://wpnews.pro/news/chasing-driver-truth-and-vram-ghosts", "canonical_source": "https://www.gladlabs.io/posts/chasing-driver-truth-and-vram-ghosts-81a68447", "published_at": "2026-08-12 02:28:05+00:00", "updated_at": "2026-08-12 03:48:26.781956+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools"], "entities": ["Glad-Labs", "Poindexter", "wan-server", "Prometheus", "MCP"], "alternates": {"html": "https://wpnews.pro/news/chasing-driver-truth-and-vram-ghosts", "markdown": "https://wpnews.pro/news/chasing-driver-truth-and-vram-ghosts.md", "text": "https://wpnews.pro/news/chasing-driver-truth-and-vram-ghosts.txt", "jsonld": "https://wpnews.pro/news/chasing-driver-truth-and-vram-ghosts.jsonld"}}