Chasing Driver Truth and VRAM Ghosts

Glad-Labs' Poindexter project shipped fixes on 2026-08-11 to stop silent animation failures by reading VRAM directly from the wan server's /health endpoint via torch.cuda.mem_get_info(0), replacing lagging Prometheus metrics. Release 0.120.0 added schedule_post for MCP and new prime-time slots, and wired up findings.<kind>.cooldown_minutes to activate 37 previously ignored seeded keys.

What we shipped on 2026-08-11 We spent most of today fighting a silent failure in the hero-plate gate, culminating in “size the hero plate from live device VRAM, not a lagging scrape” PR 3183 . We had added a quality floor to prevent generative slop, but that floor became a ceiling that silently disabled motion entirely. The first leak was logic: our VRAM check measured raw free memory while wan was already resident from a previous shot PR 3181 . The system saw 1GB free and skipped the animation, ignoring the fact that the model it needed was already occupying 23GB. We fixed this by introducing wan resident gb , which pulls vram used mb from the wan server’s /health endpoint so we can count the resident pool toward the available budget PR 3181 . Then we hit an ordering problem and a telemetry lag. We were probing VRAM at the top of animate hero , but the reclaim logic lived inside render generative clip –meaning we measured the card while image-gen was still hogging 25GB PR 3182 . Even after flipping the order, Prometheus scrape lag meant our worker was reading data that was ~40s old. We tried riding out the interval by taking the max of four samples PR 3182 , but it wasn’t enough. The final fix was to stop trusting the metrics pipeline for real-time gating. By hitting wan-server /health and pulling device free mb via torch.cuda.mem get info 0 , we get driver truth in milliseconds rather than Prometheus approximations PR 3183 . Outside the GPU wars, we cleaned up some technical debt in our testing suite. We found that tests/unit/cli/test integrations youtube cli.py was using importlib.reload on every test, which caused group identity churn and left the CLI operator holding a different group object than the module exposed PR 3173 . We also scrubbed the CI runner docs to remove “fossil” references to --forked and corrected a misleading claim about .env files that don’t actually exist in our operator stack PR 3172 . On the shipping front, release 0.120.0 landed with schedule post for MCP–allowing us to approve and publish slots in one call–and new prime-time slots so night-time publishes can be promoted in the morning PR 3146 . We also finally wired up findings.<kind .cooldown minutes PR 3169 , turning 37 seeded keys that were being ignored into actual logic. We’re moving away from relying on external telemetry for hardware gates. Driver-level truth is the only way to keep the generative pipeline tight without skipping frames. Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.