# The boring layer around your LLM call > Source: > Published: 2026-08-05 14:45:12+00:00 Most of what I learned this year came from building the interesting parts. The retrieval, the prompts, the agent loop. The stuff that's fun to think about. Then I put a FastAPI endpoint in front of one of my projects, showed it to a friend, and watched him break it in about four minutes. Not maliciously. He just asked something long and weird and the request sat there spinning until he closed the tab. That's when I realised the model was maybe a third of the actual work. The rest is the stuff wrapped around it, which is boring and nobody writes tutorials about it because it isn't interesting to build. This is what I've ended up with. I'm still a student, so treat this as notes rather than advice from someone who's run this at scale. I probably have some of it wrong. This was the first one that bit me. I assumed there was a sensible default somewhere. There sort of is, depending on your client, but it tends to be very long or effectively absent. So when a provider gets slow, your request doesn't fail. It just waits. And while it waits it's holding a worker that can't do anything else. ``` response = await litellm.acompletion( model="deepseek/deepseek-chat", messages=messages, timeout=30, ) ``` Thirty seconds felt aggressive to me at first, then I actually measured my p95 and realised nothing legitimate was taking longer than about fifteen. If a call is at thirty seconds it's already gone wrong and waiting longer doesn't help. One thing that confused me for a while: if you're behind something with its own timeout (nginx, a cloud load balancer, an API gateway) and yours is longer than theirs, you get the worst version. The user gets a 504, and your call carries on running and carries on costing money for an answer nobody will ever see. Yours should be the shorter one. Everyone tells you to retry with exponential backoff. That part's fine, most libraries do it for you. What I didn't think about was that different errors mean different things. A 429 means slow down, you're going too fast, and backing off is exactly right. A 500 means the provider had a problem and retrying is reasonable. A 400 means your request was malformed and retrying it will produce the identical error every time while you pay for the privilege of finding out. ``` response = await litellm.acompletion( model="deepseek/deepseek-chat", messages=messages, timeout=30, num_retries=2, ) ``` Two retries, not five. I had a bug at one point where a retry loop and a validation failure fed each other and the same request went out something like a dozen times before I noticed. Nothing dramatic happened because my test corpus was tiny and it was pennies, but the same shape of bug on a real workload is how you end up explaining a bill to someone. Also worth logging the requests that exhausted their retries and gave up. It's easy to only log errors that surface to the user, and the ones that quietly failed after three attempts are exactly the ones you want to know about. I knew about `max_tokens` . Everyone knows about `max_tokens` . It bounds what comes back. It took me longer to think properly about the input side, and in RAG that's where the risk actually is, because you're stuffing retrieved chunks into the prompt and you don't fully control what those chunks contain. Most of my documents were normal. One of them was enormous. It went through the same code path as everything else and cost roughly ten times what a typical request cost, and I only found it because I was staring at per-call costs for an unrelated reason. So now I count tokens before sending, not after: ``` python import tiktoken enc = tiktoken.get_encoding("cl100k_base") def cap_context(chunks, budget=6000): kept, used = [], 0 for chunk in chunks: cost = len(enc.encode(chunk)) if used + cost > budget: break kept.append(chunk) used += cost return kept ``` Crude, and it drops chunks by position rather than by relevance, which isn't ideal. But a crude ceiling you actually have beats an elegant one you're planning to add. Providers go down. Not often, but they do, and when it happens there is nothing you can do except wait, which is a bad thing to discover during a demo. LiteLLM makes this genuinely easy, which is most of why I use it: ``` response = await litellm.acompletion( model="deepseek/deepseek-chat", messages=messages, timeout=30, num_retries=2, fallbacks=["gemini/gemini-2.0-flash"], ) ``` The fallback doesn't have to be as good. That's the bit I initially misunderstood. It's the difference between a slightly worse answer and no answer at all, and users are much more forgiving of the first one. Do actually test it though. I had a fallback configured for a while that would have failed if it ever triggered, because the model name was wrong and nothing had ever exercised that path. I found it by deliberately putting a garbage primary model name in and seeing what happened, which took two minutes and I should have done it immediately. Pydantic is great. You define the shape you want, you get a clean error when the model returns something else. ``` python from pydantic import BaseModel class Answer(BaseModel): text: str confidence: float sources: list[str] ``` What tripped me up is that catching the error is only half a decision. You still have to choose what happens next, and I didn't choose for a while, which meant my choice was "throw a 500 at the user" by default. The options I've ended up thinking about: Retry once, feeding the validation error back into the prompt. This works surprisingly often for small schema mistakes. It also costs you another call and another few seconds, so it's not free. Fall back to something simpler. If the structured version keeps failing, take plain text and lose the structure rather than losing the response. Fail properly, with a real message. Sometimes this is right. But "sorry, something went wrong" is much better than a stack trace, and it's better than silently returning an empty object that breaks something three layers up. None of these is correct in general. The point is just to pick one on purpose. This is the one I'd add first if I were starting again. Provider dashboards are good. They are also retrospective. They tell you what you spent after you spent it, and if something loops overnight you find out in the morning. So I keep a counter in the process: ``` python class SpendGuard: def __init__(self, ceiling_usd): self.ceiling = ceiling_usd self.spent = 0.0 def record(self, response): cost = response._hidden_params.get("response_cost", 0) self.spent += cost if self.spent > self.ceiling: raise RuntimeError( f"Spend ceiling hit: ${self.spent:.2f} of ${self.ceiling:.2f}" ) ``` It's twenty lines and it's naive. It resets when the process restarts, and it won't help you across multiple workers unless you move the counter somewhere shared like Redis. But it turns "unbounded" into "bounded", and that's the part that actually matters. My entire dissertation experiment ran on about four pounds of compute, and knowing a bug couldn't turn that into four hundred let me iterate a lot more freely. If you're on a hosted provider, set a hard billing limit in their console too. Belt and braces. Honestly, I'd write all of this before writing any of the interesting parts. Every item here I added after something surprised me, which meant each one arrived as a small panic rather than as a decision. It's not much code. It's a timeout, a retry cap, two token ceilings, a fallback, a validation branch, and a counter. Maybe an hour to put in place at the start, versus finding each one individually the hard way. The model is the part everyone talks about. The stuff around it is what determines whether the thing survives contact with an actual user. If you're further along than me and I've got something wrong here, I'd genuinely like to know.