{"slug": "locking-the-gpu-and-silencing-the-noise", "title": "Locking the GPU and silencing the noise", "summary": "Glad-Labs' Poindexter project fixed GPU contention and silent pipeline failures on 2026-08-09, wrapping image generation in a GPU lock and requiring shot lists for media dispatch to recover 27 lost pieces. The team also triaged 4,252 GlitchTip events, finding only 9 needed human attention, and improved social promotion scheduling with prime-time slots.", "body_md": "*What we shipped on 2026-08-09*\n\nWe spent the morning figuring out why `poindexter tasks regen-image`\n\nwas taking three attempts just to get a single featured image (PR #3136). The first two tries were hitting HTTP 503s because of CUDA OOMs–the image-gen server had only 30.94 MiB free while the writer LLM sat idle on its `keep_alive`\n\n, hogging nearly 19 GB of VRAM. We realized our operator single-image endpoints via `ImageService.generate_image`\n\nwere POSTing straight to the renderer with no GPU coordination. The fix was wrapping those calls in `gpu.lock(\"image_gen\")`\n\n, which forces a call to `_unload_ollama_models()`\n\nand confirms the release via `/api/ps`\n\nbefore yielding.\n\nThat lack of strictness had leaked into other areas, most notably the media pipeline (PR #3138). We discovered 27 publishable pieces that had effectively vanished; they’d paid for TTS and transcription, but a GPU-busy skip during `generate_video_shot_list`\n\nleft them with an empty shot list. Because we’d previously decided that render nodes should “no-op gracefully” when lists were absent, these pieces just skipped rendering and retired themselves as successes. We reversed that logic–`dispatch_media_pipeline`\n\nnow requires a shot list with actual shots to proceed. We also had to thread the `short_summary_script`\n\ninto the director context during backfills (PR #3141) to ensure the 9:16 lane didn’t silently skip while reporting success.\n\nOn the ops side, we spent some time triaging a GlitchTip backlog of 4,252 events and found that only about 9 actually needed human eyes (PR #3133). The rest was mechanical noise. We had one outage fragmented into 164 separate issues because `S3UploadFailedError`\n\nwas including random tempfile names in the signature. Even worse, `GpuBusyError: GPU admission rejected: no_fit`\n\nwas being captured as an error 1,320 times, despite it being a designed outcome for fail-soft callers. We’ve now dropped that in `_before_send`\n\n, treating it like a `GraphInterrupt`\n\n.\n\nWe also tightened the social promotion logic (PR #3139). Our previous “quiet hours” implementation was too blunt; if a post went live at 11pm, every displaced promo was simply clamped to the edge of the quiet window at 07:00. This destroyed the per-platform stagger we wanted. By adding prime-time slots, we can now ensure a night publish promotes in the morning across different platforms at varied, sensible times rather than all at once. Finally, we added `schedule_post`\n\nto our MCP servers (PR #3134) so that scheduling isn’t gated behind the console or CLI, ensuring the MCP layer respects the same contract as the API–specifically checking `scheduled_for`\n\ninstead of relying on a 200 OK.\n\nWe’re moving away from “graceful no-ops” toward explicit requirements. It’s more friction upfront, but it stops pieces from silently dying in the pipeline.\n\n*Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.*", "url": "https://wpnews.pro/news/locking-the-gpu-and-silencing-the-noise", "canonical_source": "https://www.gladlabs.io/posts/locking-the-gpu-and-silencing-the-noise-ba438aee", "published_at": "2026-08-10 01:58:39+00:00", "updated_at": "2026-08-10 02:47:59.857469+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": ["Glad-Labs", "Poindexter", "GlitchTip", "ImageService", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/locking-the-gpu-and-silencing-the-noise", "markdown": "https://wpnews.pro/news/locking-the-gpu-and-silencing-the-noise.md", "text": "https://wpnews.pro/news/locking-the-gpu-and-silencing-the-noise.txt", "jsonld": "https://wpnews.pro/news/locking-the-gpu-and-silencing-the-noise.jsonld"}}