{"slug": "the-boring-layer-around-your-llm-call", "title": "The boring layer around your LLM call", "summary": "A developer building LLM applications found that the model itself is only a third of the work, with the surrounding infrastructure—timeouts, retries, and token management—proving critical. They implemented a 30-second timeout after measuring p95 latency, limited retries to two, and added input token capping to prevent oversized RAG prompts from inflating costs. The developer, still a student, shared these as notes rather than production-tested advice.", "body_md": "Most of what I learned this year came from building the interesting parts. The retrieval, the prompts, the agent loop. The stuff that's fun to think about.\n\nThen I put a FastAPI endpoint in front of one of my projects, showed it to a friend, and watched him break it in about four minutes. Not maliciously. He just asked something long and weird and the request sat there spinning until he closed the tab.\n\nThat's when I realised the model was maybe a third of the actual work. The rest is the stuff wrapped around it, which is boring and nobody writes tutorials about it because it isn't interesting to build.\n\nThis is what I've ended up with. I'm still a student, so treat this as notes rather than advice from someone who's run this at scale. I probably have some of it wrong.\n\nThis was the first one that bit me.\n\nI assumed there was a sensible default somewhere. There sort of is, depending on your client, but it tends to be very long or effectively absent. So when a provider gets slow, your request doesn't fail. It just waits. And while it waits it's holding a worker that can't do anything else.\n\n```\nresponse = await litellm.acompletion(\n    model=\"deepseek/deepseek-chat\",\n    messages=messages,\n    timeout=30,\n)\n```\n\nThirty seconds felt aggressive to me at first, then I actually measured my p95 and realised nothing legitimate was taking longer than about fifteen. If a call is at thirty seconds it's already gone wrong and waiting longer doesn't help.\n\nOne thing that confused me for a while: if you're behind something with its own timeout (nginx, a cloud load balancer, an API gateway) and yours is longer than theirs, you get the worst version. The user gets a 504, and your call carries on running and carries on costing money for an answer nobody will ever see. Yours should be the shorter one.\n\nEveryone tells you to retry with exponential backoff. That part's fine, most libraries do it for you.\n\nWhat I didn't think about was that different errors mean different things.\n\nA 429 means slow down, you're going too fast, and backing off is exactly right. A 500 means the provider had a problem and retrying is reasonable. A 400 means your request was malformed and retrying it will produce the identical error every time while you pay for the privilege of finding out.\n\n```\nresponse = await litellm.acompletion(\n    model=\"deepseek/deepseek-chat\",\n    messages=messages,\n    timeout=30,\n    num_retries=2,\n)\n```\n\nTwo retries, not five. I had a bug at one point where a retry loop and a validation failure fed each other and the same request went out something like a dozen times before I noticed. Nothing dramatic happened because my test corpus was tiny and it was pennies, but the same shape of bug on a real workload is how you end up explaining a bill to someone.\n\nAlso worth logging the requests that exhausted their retries and gave up. It's easy to only log errors that surface to the user, and the ones that quietly failed after three attempts are exactly the ones you want to know about.\n\nI knew about `max_tokens`\n\n. Everyone knows about `max_tokens`\n\n. It bounds what comes back.\n\nIt took me longer to think properly about the input side, and in RAG that's where the risk actually is, because you're stuffing retrieved chunks into the prompt and you don't fully control what those chunks contain. Most of my documents were normal. One of them was enormous. It went through the same code path as everything else and cost roughly ten times what a typical request cost, and I only found it because I was staring at per-call costs for an unrelated reason.\n\nSo now I count tokens before sending, not after:\n\n``` python\nimport tiktoken\n\nenc = tiktoken.get_encoding(\"cl100k_base\")\n\ndef cap_context(chunks, budget=6000):\n    kept, used = [], 0\n    for chunk in chunks:\n        cost = len(enc.encode(chunk))\n        if used + cost > budget:\n            break\n        kept.append(chunk)\n        used += cost\n    return kept\n```\n\nCrude, and it drops chunks by position rather than by relevance, which isn't ideal. But a crude ceiling you actually have beats an elegant one you're planning to add.\n\nProviders go down. Not often, but they do, and when it happens there is nothing you can do except wait, which is a bad thing to discover during a demo.\n\nLiteLLM makes this genuinely easy, which is most of why I use it:\n\n```\nresponse = await litellm.acompletion(\n    model=\"deepseek/deepseek-chat\",\n    messages=messages,\n    timeout=30,\n    num_retries=2,\n    fallbacks=[\"gemini/gemini-2.0-flash\"],\n)\n```\n\nThe fallback doesn't have to be as good. That's the bit I initially misunderstood. It's the difference between a slightly worse answer and no answer at all, and users are much more forgiving of the first one.\n\nDo actually test it though. I had a fallback configured for a while that would have failed if it ever triggered, because the model name was wrong and nothing had ever exercised that path. I found it by deliberately putting a garbage primary model name in and seeing what happened, which took two minutes and I should have done it immediately.\n\nPydantic is great. You define the shape you want, you get a clean error when the model returns something else.\n\n``` python\nfrom pydantic import BaseModel\n\nclass Answer(BaseModel):\n    text: str\n    confidence: float\n    sources: list[str]\n```\n\nWhat tripped me up is that catching the error is only half a decision. You still have to choose what happens next, and I didn't choose for a while, which meant my choice was \"throw a 500 at the user\" by default.\n\nThe options I've ended up thinking about:\n\nRetry once, feeding the validation error back into the prompt. This works surprisingly often for small schema mistakes. It also costs you another call and another few seconds, so it's not free.\n\nFall back to something simpler. If the structured version keeps failing, take plain text and lose the structure rather than losing the response.\n\nFail properly, with a real message. Sometimes this is right. But \"sorry, something went wrong\" is much better than a stack trace, and it's better than silently returning an empty object that breaks something three layers up.\n\nNone of these is correct in general. The point is just to pick one on purpose.\n\nThis is the one I'd add first if I were starting again.\n\nProvider dashboards are good. They are also retrospective. They tell you what you spent after you spent it, and if something loops overnight you find out in the morning.\n\nSo I keep a counter in the process:\n\n``` python\nclass SpendGuard:\n    def __init__(self, ceiling_usd):\n        self.ceiling = ceiling_usd\n        self.spent = 0.0\n\n    def record(self, response):\n        cost = response._hidden_params.get(\"response_cost\", 0)\n        self.spent += cost\n        if self.spent > self.ceiling:\n            raise RuntimeError(\n                f\"Spend ceiling hit: ${self.spent:.2f} of ${self.ceiling:.2f}\"\n            )\n```\n\nIt's twenty lines and it's naive. It resets when the process restarts, and it won't help you across multiple workers unless you move the counter somewhere shared like Redis. But it turns \"unbounded\" into \"bounded\", and that's the part that actually matters. My entire dissertation experiment ran on about four pounds of compute, and knowing a bug couldn't turn that into four hundred let me iterate a lot more freely.\n\nIf you're on a hosted provider, set a hard billing limit in their console too. Belt and braces.\n\nHonestly, I'd write all of this before writing any of the interesting parts.\n\nEvery item here I added after something surprised me, which meant each one arrived as a small panic rather than as a decision. It's not much code. It's a timeout, a retry cap, two token ceilings, a fallback, a validation branch, and a counter. Maybe an hour to put in place at the start, versus finding each one individually the hard way.\n\nThe model is the part everyone talks about. The stuff around it is what determines whether the thing survives contact with an actual user.\n\nIf you're further along than me and I've got something wrong here, I'd genuinely like to know.", "url": "https://wpnews.pro/news/the-boring-layer-around-your-llm-call", "canonical_source": "https://dev.to/gsatya147/the-boring-layer-around-your-llm-call-47nn", "published_at": "2026-08-05 14:45:12+00:00", "updated_at": "2026-08-05 15:01:04.221570+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["FastAPI", "LiteLLM", "DeepSeek", "tiktoken"], "alternates": {"html": "https://wpnews.pro/news/the-boring-layer-around-your-llm-call", "markdown": "https://wpnews.pro/news/the-boring-layer-around-your-llm-call.md", "text": "https://wpnews.pro/news/the-boring-layer-around-your-llm-call.txt", "jsonld": "https://wpnews.pro/news/the-boring-layer-around-your-llm-call.jsonld"}}