cd /news/ai-infrastructure/chasing-driver-truth-and-vram-ghosts · home topics ai-infrastructure article
[ARTICLE · art-92983] src=gladlabs.io ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Chasing Driver Truth and VRAM Ghosts

Glad-Labs' Poindexter project shipped fixes on 2026-08-11 to stop silent animation failures by reading VRAM directly from the wan server's /health endpoint via torch.cuda.mem_get_info(0), replacing lagging Prometheus metrics. Release 0.120.0 added schedule_post for MCP and new prime-time slots, and wired up findings.<kind>.cooldown_minutes to activate 37 previously ignored seeded keys.

read2 min views1 publishedAug 12, 2026
Chasing Driver Truth and VRAM Ghosts
Image: Gladlabs (auto-discovered)

What we shipped on 2026-08-11 We spent most of today fighting a silent failure in the hero-plate gate, culminating in “size the hero plate from live device VRAM, not a lagging scrape” (PR #3183). We had added a quality floor to prevent generative slop, but that floor became a ceiling that silently disabled motion entirely.

The first leak was logic: our VRAM check measured raw free memory while wan

was already resident from a previous shot (PR #3181). The system saw 1GB free and skipped the animation, ignoring the fact that the model it needed was already occupying 23GB. We fixed this by introducing _wan_resident_gb()

, which pulls vram_used_mb

from the wan server’s /health endpoint so we can count the resident pool toward the available budget (PR #3181).

Then we hit an ordering problem and a telemetry lag. We were probing VRAM at the top of _animate_hero

, but the reclaim logic lived inside _render_generative_clip

–meaning we measured the card while image-gen

was still hogging 25GB (PR #3182). Even after flipping the order, Prometheus scrape lag meant our worker was reading data that was ~40s old. We tried riding out the interval by taking the max of four samples (PR #3182), but it wasn’t enough.

The final fix was to stop trusting the metrics pipeline for real-time gating. By hitting wan-server /health

and pulling device_free_mb

via torch.cuda.mem_get_info(0) , we get driver truth in milliseconds rather than Prometheus approximations (PR #3183).

Outside the GPU wars, we cleaned up some technical debt in our testing suite. We found that tests/unit/cli/test_integrations_youtube_cli.py

was using importlib.reload

on every test, which caused group identity churn and left the CLI operator holding a different group object than the module exposed (PR #3173). We also scrubbed the CI runner docs to remove “fossil” references to --forked

and corrected a misleading claim about .env

files that don’t actually exist in our operator stack (PR #3172).

On the shipping front, release 0.120.0 landed with schedule_post

for MCP–allowing us to approve and publish slots in one call–and new prime-time slots so night-time publishes can be promoted in the morning (PR #3146). We also finally wired up findings.<kind>.cooldown_minutes (PR #3169), turning 37 seeded keys that were being ignored into actual logic.

We’re moving away from relying on external telemetry for hardware gates. Driver-level truth is the only way to keep the generative pipeline tight without skipping frames.

Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @glad-labs 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/chasing-driver-truth…] indexed:0 read:2min 2026-08-12 ·