# Locking the GPU and silencing the noise

> Source: <https://www.gladlabs.io/posts/locking-the-gpu-and-silencing-the-noise-ba438aee>
> Published: 2026-08-10 01:58:39+00:00

*What we shipped on 2026-08-09*

We spent the morning figuring out why `poindexter tasks regen-image`

was taking three attempts just to get a single featured image (PR #3136). The first two tries were hitting HTTP 503s because of CUDA OOMs–the image-gen server had only 30.94 MiB free while the writer LLM sat idle on its `keep_alive`

, hogging nearly 19 GB of VRAM. We realized our operator single-image endpoints via `ImageService.generate_image`

were POSTing straight to the renderer with no GPU coordination. The fix was wrapping those calls in `gpu.lock("image_gen")`

, which forces a call to `_unload_ollama_models()`

and confirms the release via `/api/ps`

before yielding.

That lack of strictness had leaked into other areas, most notably the media pipeline (PR #3138). We discovered 27 publishable pieces that had effectively vanished; they’d paid for TTS and transcription, but a GPU-busy skip during `generate_video_shot_list`

left them with an empty shot list. Because we’d previously decided that render nodes should “no-op gracefully” when lists were absent, these pieces just skipped rendering and retired themselves as successes. We reversed that logic–`dispatch_media_pipeline`

now requires a shot list with actual shots to proceed. We also had to thread the `short_summary_script`

into the director context during backfills (PR #3141) to ensure the 9:16 lane didn’t silently skip while reporting success.

On the ops side, we spent some time triaging a GlitchTip backlog of 4,252 events and found that only about 9 actually needed human eyes (PR #3133). The rest was mechanical noise. We had one outage fragmented into 164 separate issues because `S3UploadFailedError`

was including random tempfile names in the signature. Even worse, `GpuBusyError: GPU admission rejected: no_fit`

was being captured as an error 1,320 times, despite it being a designed outcome for fail-soft callers. We’ve now dropped that in `_before_send`

, treating it like a `GraphInterrupt`

.

We also tightened the social promotion logic (PR #3139). Our previous “quiet hours” implementation was too blunt; if a post went live at 11pm, every displaced promo was simply clamped to the edge of the quiet window at 07:00. This destroyed the per-platform stagger we wanted. By adding prime-time slots, we can now ensure a night publish promotes in the morning across different platforms at varied, sensible times rather than all at once. Finally, we added `schedule_post`

to our MCP servers (PR #3134) so that scheduling isn’t gated behind the console or CLI, ensuring the MCP layer respects the same contract as the API–specifically checking `scheduled_for`

instead of relying on a 200 OK.

We’re moving away from “graceful no-ops” toward explicit requirements. It’s more friction upfront, but it stops pieces from silently dying in the pipeline.

*Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.*