# Chasing Driver Truth and VRAM Ghosts

> Source: <https://www.gladlabs.io/posts/chasing-driver-truth-and-vram-ghosts-81a68447>
> Published: 2026-08-12 02:28:05+00:00

*What we shipped on 2026-08-11*

We spent most of today fighting a silent failure in the hero-plate gate, culminating in “size the hero plate from live device VRAM, not a lagging scrape” (PR #3183). We had added a quality floor to prevent generative slop, but that floor became a ceiling that silently disabled motion entirely.

The first leak was logic: our VRAM check measured raw free memory while `wan`

was already resident from a previous shot (PR #3181). The system saw 1GB free and skipped the animation, ignoring the fact that the model it needed was already occupying 23GB. We fixed this by introducing `_wan_resident_gb()`

, which pulls `vram_used_mb`

from the wan server’s `/health`

endpoint so we can count the resident pool toward the available budget (PR #3181).

Then we hit an ordering problem and a telemetry lag. We were probing VRAM at the top of `_animate_hero`

, but the reclaim logic lived inside `_render_generative_clip`

–meaning we measured the card while `image-gen`

was still hogging 25GB (PR #3182). Even after flipping the order, Prometheus scrape lag meant our worker was reading data that was ~40s old. We tried riding out the interval by taking the max of four samples (PR #3182), but it wasn’t enough.

The final fix was to stop trusting the metrics pipeline for real-time gating. By hitting `wan-server /health`

and pulling `device_free_mb`

via `torch.cuda.mem_get_info(0)`

, we get driver truth in milliseconds rather than Prometheus approximations (PR #3183).

Outside the GPU wars, we cleaned up some technical debt in our testing suite. We found that `tests/unit/cli/test_integrations_youtube_cli.py`

was using `importlib.reload`

on every test, which caused group identity churn and left the CLI operator holding a different group object than the module exposed (PR #3173). We also scrubbed the CI runner docs to remove “fossil” references to `--forked`

and corrected a misleading claim about `.env`

files that don’t actually exist in our operator stack (PR #3172).

On the shipping front, release 0.120.0 landed with `schedule_post`

for MCP–allowing us to approve and publish slots in one call–and new prime-time slots so night-time publishes can be promoted in the morning (PR #3146). We also finally wired up `findings.<kind>.cooldown_minutes`

(PR #3169), turning 37 seeded keys that were being ignored into actual logic.

We’re moving away from relying on external telemetry for hardware gates. Driver-level truth is the only way to keep the generative pipeline tight without skipping frames.

*Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.*