cd /news/ai-infrastructure/locking-the-gpu-and-silencing-the-no… · home topics ai-infrastructure article
[ARTICLE · art-89776] src=gladlabs.io ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Locking the GPU and silencing the noise

Glad-Labs' Poindexter project fixed GPU contention and silent pipeline failures on 2026-08-09, wrapping image generation in a GPU lock and requiring shot lists for media dispatch to recover 27 lost pieces. The team also triaged 4,252 GlitchTip events, finding only 9 needed human attention, and improved social promotion scheduling with prime-time slots.

read2 min views1 publishedAug 10, 2026
Locking the GPU and silencing the noise
Image: Gladlabs (auto-discovered)

What we shipped on 2026-08-09 We spent the morning figuring out why poindexter tasks regen-image

was taking three attempts just to get a single featured image (PR #3136). The first two tries were hitting HTTP 503s because of CUDA OOMs–the image-gen server had only 30.94 MiB free while the writer LLM sat idle on its keep_alive

, hogging nearly 19 GB of VRAM. We realized our operator single-image endpoints via ImageService.generate_image

were POSTing straight to the renderer with no GPU coordination. The fix was wrapping those calls in gpu.lock("image_gen")

, which forces a call to _unload_ollama_models()

and confirms the release via /api/ps

before yielding.

That lack of strictness had leaked into other areas, most notably the media pipeline (PR #3138). We discovered 27 publishable pieces that had effectively vanished; they’d paid for TTS and transcription, but a GPU-busy skip during generate_video_shot_list

left them with an empty shot list. Because we’d previously decided that render nodes should “no-op gracefully” when lists were absent, these pieces just skipped rendering and retired themselves as successes. We reversed that logic–dispatch_media_pipeline

now requires a shot list with actual shots to proceed. We also had to thread the short_summary_script

into the director context during backfills (PR #3141) to ensure the 9:16 lane didn’t silently skip while reporting success.

On the ops side, we spent some time triaging a GlitchTip backlog of 4,252 events and found that only about 9 actually needed human eyes (PR #3133). The rest was mechanical noise. We had one outage fragmented into 164 separate issues because S3UploadFailedError

was including random tempfile names in the signature. Even worse, GpuBusyError: GPU admission rejected: no_fit

was being captured as an error 1,320 times, despite it being a designed outcome for fail-soft callers. We’ve now dropped that in _before_send

, treating it like a GraphInterrupt

.

We also tightened the social promotion logic (PR #3139). Our previous “quiet hours” implementation was too blunt; if a post went live at 11pm, every displaced promo was simply clamped to the edge of the quiet window at 07:00. This destroyed the per-platform stagger we wanted. By adding prime-time slots, we can now ensure a night publish promotes in the morning across different platforms at varied, sensible times rather than all at once. Finally, we added schedule_post

to our MCP servers (PR #3134) so that scheduling isn’t gated behind the console or CLI, ensuring the MCP layer respects the same contract as the API–specifically checking scheduled_for

instead of relying on a 200 OK.

We’re moving away from “graceful no-ops” toward explicit requirements. It’s more friction upfront, but it stops pieces from silently dying in the pipeline.

Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @glad-labs 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/locking-the-gpu-and-…] indexed:0 read:2min 2026-08-10 ·