{"slug": "the-retry-that-wasn-t-the-fix", "title": "The Retry That Wasn't the Fix", "summary": "A developer building an autonomous software-engineering agent traced intermittent 503 errors from NVIDIA's hosted API to the Python SDK's structured-output path, which silently falls through multiple structured-output strategies and issues two or three round-trips per call instead of one. Bypassing the SDK with raw HTTP logging revealed the errors were concentrated on requests triggering the multi-attempt cascade, not random provider flakiness.", "body_md": "Every engineer knows this feeling: a request fails, you retry it, it works, and you tell yourself you'll come back to figure out why later. Later rarely comes. The retry quietly becomes the fix, and the real cause gets buried under a try/except block nobody reads again.\n\nI almost did exactly that. This is the story of the one time I didn't, and what I found when I refused to.\n\nI was building an autonomous software-engineering agent — a system that takes a task, plans it, writes a code patch, and asks an LLM to return that patch in a strict, structured format so my code can parse it reliably. The LLM provider was NVIDIA's hosted API, and the request type was \"structured output\" — a fancy way of saying: don't just talk to me, fill out this exact JSON shape.\n\nMost of the time, it worked fine. But every so often — not on a schedule, not tied to load, not reproducible on demand — a request would come back with a `503`. Service unavailable. No pattern I could see. It happened maybe once every fifteen or twenty calls, always on this one request type, never on simpler unstructured requests to the same provider.\n\nThe obvious move was to wrap the call in a retry with backoff and call it a day. It would have worked, in the sense that the errors would have mostly disappeared from the logs. It also would have told me nothing, and it would have doubled the latency and cost of every request that happened to hit the failure — forever, without me ever knowing why.\n\nA retry that works is not the same as a bug that's understood. One hides the problem. The other removes it.\n\nI'd already built real fallback logic elsewhere in the system — if a provider genuinely goes down, switch to a backup. But fallback is for provider outages, not for masking a bug I hadn't diagnosed yet. Using it here would have meant permanently paying a tax for a problem I'd never actually looked at.\n\nThe request went through a Python SDK, which went through a LangChain integration, which went through several layers before it ever touched the network. My first pass was reading through that code, adding print statements, trying to catch the 503 in the act. It didn't work. The failure was too intermittent to catch by staring at logs — I'd add logging, run it a dozen times, get nothing, and start doubting whether there was a real pattern at all.\n\nHonestly, I gave up on it for three days. Not officially — I just stopped opening that file and worked on other parts of the system instead, telling myself I'd come back to it. That's a normal thing to do and also, in hindsight, not a bad instinct — some problems need you to stop staring directly at them.\n\nWhen I came back to it on day four, I stopped trying to read the SDK's code and decided to just watch what it was actually doing on the wire. I bypassed the SDK entirely and made the exact same request with raw HTTP calls, logging everything myself — headers, payload, timing, response. No library between me and what was actually happening.\n\nThat's when the pattern showed up almost immediately. The SDK's structured-output path wasn't sending one request per call. Depending on how the response format was specified, it could fall through several different structured-output strategies in sequence — try one, and if the shape didn't quite match what it expected, try another. What looked like one API call from my code was sometimes two or three round-trips against NVIDIA's servers, and I'd never have seen that from inside the SDK's own logs, because the SDK doesn't log its own indecision.\n\n**What this told me:** the 503s weren't random. They were concentrated on exactly the requests that triggered this multi-attempt cascade — which meant the problem wasn't \"NVIDIA is flaky,\" it was \"I'm accidentally hammering one specific endpoint shape harder than I think I am.\"\n\nThe SDK's default structured-output mechanism was built to mimic OpenAI's `response_format` convention, which isn't native to NVIDIA's API — it gets translated underneath, and that translation was where the extra round-trips were coming from. NVIDIA's own models support a native mechanism called `guided_json`, which does the same job — force the output into a schema — without going through that compatibility shim at all.\n\nI switched the request to use `guided_json` directly. One request per call, no cascade, no silent retries I didn't ask for. The 503s didn't get retried away — they stopped happening, because the thing that was causing them stopped happening.\n\nI keep coming back to that part when I think about this bug, more than the fix itself. It's tempting to write this up like I sat down, was clever, and solved it in one sitting — that's how most of these posts get written, and it's usually not quite how it happened. What actually happened is I got stuck, got annoyed, and walked away from the file for three days. The thing that made progress possible when I came back wasn't a burst of cleverness. It was just refusing to accept \"it's flaky, whatever\" as the final answer once I sat back down.\n\nTwo things stuck with me. A \"random\" bug is almost always a bug where you haven't found the variable it depends on yet — every intermittent failure I've ever actually run down had a trigger, once I stopped assuming there wasn't one. And whatever library sits between your code and the actual network call exists to make the common case easy, which means it's usually the first place hiding the uncommon case you're trying to see. Going around it — raw HTTP, or whatever \"raw\" means for your stack — turned out to be a lot faster than reading someone else's source code trying to guess what it does under load.\n\nThe actual fix was a few lines of configuration. Finding it cost three days I didn't spend on it, and maybe twenty minutes once I stopped reading code and started reading traffic instead.\n\nI'm a Data Science student at IIT Madras, building autonomous, security-hardened AI agent systems — this one being an agentic software-engineering pipeline with human-approval gates before any code it writes gets committed. If you're working on something similar, or hiring for GenAI/agentic AI roles, I'd like to talk — [github.com/aman-kumar1905](https://github.com/aman-kumar1905).", "url": "https://wpnews.pro/news/the-retry-that-wasn-t-the-fix", "canonical_source": "https://dev.to/amankumar1905/the-retry-that-wasnt-the-fix-343o", "published_at": "2026-09-26 04:04:48+00:00", "updated_at": "2026-09-26 04:59:58.305863+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "large-language-models"], "entities": ["NVIDIA", "LangChain", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-retry-that-wasn-t-the-fix", "markdown": "https://wpnews.pro/news/the-retry-that-wasn-t-the-fix.md", "text": "https://wpnews.pro/news/the-retry-that-wasn-t-the-fix.txt", "jsonld": "https://wpnews.pro/news/the-retry-that-wasn-t-the-fix.jsonld"}}