Locking the GPU and silencing the noise

Glad-Labs' Poindexter project fixed GPU contention and silent pipeline failures on 2026-08-09, wrapping image generation in a GPU lock and requiring shot lists for media dispatch to recover 27 lost pieces. The team also triaged 4,252 GlitchTip events, finding only 9 needed human attention, and improved social promotion scheduling with prime-time slots.

What we shipped on 2026-08-09 We spent the morning figuring out why poindexter tasks regen-image was taking three attempts just to get a single featured image PR 3136 . The first two tries were hitting HTTP 503s because of CUDA OOMs–the image-gen server had only 30.94 MiB free while the writer LLM sat idle on its keep alive , hogging nearly 19 GB of VRAM. We realized our operator single-image endpoints via ImageService.generate image were POSTing straight to the renderer with no GPU coordination. The fix was wrapping those calls in gpu.lock "image gen" , which forces a call to unload ollama models and confirms the release via /api/ps before yielding. That lack of strictness had leaked into other areas, most notably the media pipeline PR 3138 . We discovered 27 publishable pieces that had effectively vanished; they’d paid for TTS and transcription, but a GPU-busy skip during generate video shot list left them with an empty shot list. Because we’d previously decided that render nodes should “no-op gracefully” when lists were absent, these pieces just skipped rendering and retired themselves as successes. We reversed that logic– dispatch media pipeline now requires a shot list with actual shots to proceed. We also had to thread the short summary script into the director context during backfills PR 3141 to ensure the 9:16 lane didn’t silently skip while reporting success. On the ops side, we spent some time triaging a GlitchTip backlog of 4,252 events and found that only about 9 actually needed human eyes PR 3133 . The rest was mechanical noise. We had one outage fragmented into 164 separate issues because S3UploadFailedError was including random tempfile names in the signature. Even worse, GpuBusyError: GPU admission rejected: no fit was being captured as an error 1,320 times, despite it being a designed outcome for fail-soft callers. We’ve now dropped that in before send , treating it like a GraphInterrupt . We also tightened the social promotion logic PR 3139 . Our previous “quiet hours” implementation was too blunt; if a post went live at 11pm, every displaced promo was simply clamped to the edge of the quiet window at 07:00. This destroyed the per-platform stagger we wanted. By adding prime-time slots, we can now ensure a night publish promotes in the morning across different platforms at varied, sensible times rather than all at once. Finally, we added schedule post to our MCP servers PR 3134 so that scheduling isn’t gated behind the console or CLI, ensuring the MCP layer respects the same contract as the API–specifically checking scheduled for instead of relying on a 200 OK. We’re moving away from “graceful no-ops” toward explicit requirements. It’s more friction upfront, but it stops pieces from silently dying in the pipeline. Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.