# How a GPU Lock Bug Was Quietly Wrecking Our RAG Sweep > Source: > Published: 2026-09-03 17:43:24+00:00 We run a sweep. Every cycle, a job walks through a queue of documents, hits the GPU, pulls context, writes results, moves to the next one. Simple in theory. In practice, it’s a fight over a shared resource that doesn’t want to share. That’s what “sweep process optimization” means to us on a Tuesday afternoon. Not the elegant math version. The version where a pipeline validation run on 2026-06-19 exposed a GPU lock bug that had been quietly wrecking our internal RAG sweep, and we spent the day exorcising it (see [Fixing the GPU lock and taming the internal RAG sweep](/posts/fixing-the-gpu-lock-and-taming-the-internal-rag-sw-8d56383c)). ### What a sweep actually is A sweep process, stripped down, is a loop that has to stay inside a boundary that keeps moving. In our case, the boundary was GPU availability. The sweep job assumed the resource would be there when it asked. It wasn’t always there. Other processes were holding the lock, releasing it late, or not releasing it at all. The sweep didn’t fail loudly – it just slowed, stalled, backed up, and started producing garbage timing under load. That’s not a coincidence of naming. There’s a whole branch of control theory built around exactly this shape of problem. A sweeping process, first studied by Moreau in the 1970s, describes a point that has to stay inside a set that’s constantly in motion – and the control problem is figuring out how to steer that point without letting it get shoved outside the boundary when the set moves. Uncontrolled versions have been around for decades; the controlled case, where you actually get to influence the moving set, is newer and has drawn serious attention from applied researchers in the last several years. ### Why the math matters for a pipeline, not just a proof You don’t need to solve a variational inequality to fix a lock bug. But the framing is useful. Our GPU lock issue was, functionally, a perturbed constraint. The “safe set” for the sweep – GPU free, memory available, no contention – kept shifting because of other jobs on the box. When we treated the lock as static and just retried on failure, the sweep degraded unpredictably. Once we treated GPU availability as a moving boundary and built the sweep to track it explicitly – checking state before committing, backing off cleanly, releasing early – the whole thing stabilized. That maps onto research on [perturbed sweeping processes](https://arxiv.org/html/2407.18469v1), where the moving set isn’t clean – it’s noisy, disturbed, reacting to outside forces. The convergence analysis in that line of work exists precisely because real systems don’t get a tidy, predictable constraint. Ours didn’t either. Our cadvisor leak and the OOM cascade that nearly took down the WSL2 VM was the same pattern wearing a different hat: a resource boundary moving under us while a loop kept assuming it was fixed (see [Taming the cadvisor leak and cleaning up LLM garbage](/posts/taming-the-cadvisor-leak-and-cleaning-up-llm-garba-3361e7c5)). ### The practical version If you’re running any kind of sweep – a batch inference loop, a training scheduler, a scraper hammering a rate-limited API – assume your constraint set moves. Don’t hardcode “GPU is free” or “rate limit resets every 60 seconds” as gospel. Check state at the point of contact. Build in backoff that reacts to the actual boundary, not the boundary you expected an hour ago. The formal theory, including work presented by [Boris Mordukhovich on optimal control of perturbed sweeping processes](https://www.youtube.com/watch?v=0QmbFM0qyho), exists because this is a hard problem even with clean math. Our version had cadvisor logs and a GPU that didn’t want to let go. Different mess, same shape. Optimize the sweep by respecting the boundary, not by pretending it holds still.