How to set an AI agent tool call timeout before one slow API stalls a run Python's requests library ships with no default timeout and the OpenAI and Anthropic Python SDKs both default to a ten-minute request timeout, so a single slow tool call can hold an agent run for that long, according to a Startup Fortune guide on setting AI agent tool call timeouts. The guide recommends setting each tool's read timeout from its own p95 or p99 latency in traces, giving every run a total deadline that caps each call at the remaining time minus a reserve for the final answer, counting retries across the SDK, wrapper and agent loop, and returning a structured error to the model while cancelling the in-flight request on timeout. It notes that a stalled call does not burn tokens while waiting, but the eventual error triggers a retry that resends the full conversation context, and that ten stuck runs can occupy ten slots in a fixed worker pool. Somewhere in your agent's tool list is an API that now and then takes four minutes to answer, and nothing in your code tells the agent to stop waiting. Three numbers per tool and one deadline per run will fix that. - Python's requests library has no default timeout at all, and the OpenAI and Anthropic SDKs both default to ten minutes, so a stalled tool call can hold a run for that long. - Set each tool's read timeout from its own p95 or p99 latency in your traces, not from one shared number. - Give every run a total deadline and cap each tool call at whatever time remains, minus a reserve for the final answer. - Count retries across every layer, because SDK, wrapper and agent loop retries multiply, and only retry calls that are safe to repeat. - On timeout, hand the model a structured error it can act on, and cancel the request that's still in flight. A hung call is the quietest way for an agent to fail. There's no stack trace and no red dashboard, just a run that never finishes, a worker held open, and a user staring at a spinner. Getting an ai agent tool call timeout right means deciding, tool by tool, how long you're willing to wait, and then deciding how long the whole task may take. Most teams do neither. They inherit whatever number the HTTP client shipped with. Look at what those defaults are. The Python requests library has no timeout unless you pass one, and its own documentation says production code should always set it. The httpx client defaults to five seconds, which sounds safe until you notice that it applies to each phase of the request separately. The OpenAI and Anthropic Python SDKs both default to a ten minute request timeout, which makes sense for a long model generation and makes no sense for a tool that looks up an order status. Whichever library your tool wrapper uses, the default was chosen by someone who had never seen your agent. The first thing to get straight is where the money goes. A call that's simply waiting doesn't burn tokens. The damage comes after, when something finally breaks the wait. The request errors out at minute ten, the agent loop retries, the model re-reads the entire conversation to decide what to do next, and the same slow tool gets called again. Each pass resends the full context. A stall that looks like one problem turns into several full-price model calls plus a worker that sat idle the whole time. It also hurts in ways your cost dashboard won't show. If you run agents on a fixed pool of workers, ten stuck runs can occupy ten slots while healthy requests queue behind them. The failure spreads sideways. This is why the fix for an ai agent hanging on an api call isn't a bigger worker pool or a retry decorator. It's a budget. How to handle LLM API rate limits in production before retries burn your quota https://startupfortune.com/how-to-handle-llm-api-rate-limits-in-production-before-retries-burn-your-quota/ A 400,000 tokens-per-minute cap, minus 20% held back as headroom, leaves 320,000 to plan against. Splitting that into lanes puts 96,000 on user-facing calls, and a background step costing 18,000 tokens supports only about twelve steps a minute. - how to handle LLM API rate limits https://startupfortune.com/how-to-handle-llm-api-rate-limits-in-production-before-retries-burn-your-quota/ - prevent retries from burning API quota https://startupfortune.com/how-to-handle-llm-api-rate-limits-in-production-before-retries-burn-your-quota/ It's also different from the controls you may already have. A rate limit protects the provider from you. A max-iterations cap stops an agent that's looping. Fallback routing picks another model or tool when one is unavailable. None of them stops a single call from taking forever, because a call that never returns never hits your iteration count and never trips a rate limit. You need a clock. How to set timeout for LLM agent tool calls from real latency Start with data you almost certainly already have. If your agent writes traces, every tool call has a start and end time. Pull the last two weeks, group by tool name, and compute the p50, p95 and p99 durations. If you don't have traces, add timing around your tool wrapper today and come back in a week. Guessing is how you end up with thirty seconds everywhere. Then pick a percentile per tool. Amazon's Builders' Library article on timeouts, retries and backoff with jitter gives the cleanest rule: choose the timeout by deciding what rate of false timeouts you can live with, and read it off the latency distribution. If you can tolerate one call in a hundred being cut off when it would have succeeded, that's your p99. For tools in the middle of an interactive run, p95 plus some headroom is a sensible start, because a user would rather see a fast, honest failure than a slow success. Jeff Dean and Luiz André Barroso made the underlying point in their 2013 paper The Tail at Scale: the slowest few percent of calls behave nothing like the rest, and waiting patiently for them is how latency compounds. Here's what that looks like with plausible numbers. A product search tool shows a p50 of 400 milliseconds, a p95 of 1.8 seconds and a p99 of 6 seconds. A 3 second timeout will cut off somewhere between one and five percent of calls, and most of those retries will come back fast. A PDF export tool shows a p95 of 25 seconds, and giving it 3 seconds would break it. Two tools, two budgets. A single shared number will be wrong for at least one of them, and usually wrong for both. Frankly, the most common mistake here isn't picking a bad number. It's picking one number. The tool that takes 400 milliseconds and the one that renders a report shouldn't share a setting, and neither should a read that's safe to repeat and a write that isn't. Split the timeout into connect, read and total A single timeout value hides three different failures. A connect timeout covers how long you'll wait to open the connection, and it should be short, one or two seconds, because a server that can't accept you quickly isn't going to be helpful later. A read timeout covers silence once the request is sent. A total timeout covers the whole exchange from first byte sent to last byte received. The distinction matters most for tools that stream. A server sending a large result slowly will keep resetting a read timeout and can still run for minutes. Set a total cap as well. For streaming tools, add an idle timeout, which means you only give up if no data has arrived for a set number of seconds, and keep the total cap as a backstop. In httpx you can set connect, read, write and pool timeouts individually, and in Python 3.11 and later the asyncio.timeout context manager lets you wrap the entire tool call, including any parsing, in one hard limit. Use both. The library timeout handles the network, and the outer one handles everything else, including the case where your own code hangs. How to set max iterations for AI agent loop before it drains your budget https://startupfortune.com/how-to-set-max-iterations-for-ai-agent-loop-before-it-drains-your-budget/ In the worked example, one uncapped loop at 200 steps costs roughly $95, against about $0.34 for a healthy task, because every step resends the whole history. A dollar ceiling near $3 would have stopped that run, and sub-agents should share the parent's budget. - how to set max AI agent iterations https://startupfortune.com/how-to-set-max-iterations-for-ai-agent-loop-before-it-drains-your-budget/ - stop AI agent loops draining budget https://startupfortune.com/how-to-set-max-iterations-for-ai-agent-loop-before-it-drains-your-budget/ One more habit: store these numbers next to the tool definition, not scattered through the code. A small config entry per tool, with connect, read and total values and a note on where the number came from, lets the next engineer see that 6 seconds was a p99 from October and not a guess. Give the whole run a deadline Per-tool timeouts alone still allow a bad run. Five tool calls that each take nearly their full timeout, plus the model calls between them, can add up to a task that runs for minutes even though no single piece misbehaved. So set a deadline for the run as a whole, as a fixed point in time rather than a counter. When the run starts, record start plus budget, say 90 seconds for a support-chat task or 10 minutes for a background research job. Before every tool call, compute the time remaining. The timeout for that call becomes the smaller of the tool's own limit and the remaining time, minus a reserve. The reserve matters more than it looks. The final model call that writes the answer takes real time too, often several seconds, so if you let tools consume the entire budget you'll have nothing left to tell the user what happened. Hold back enough for that last step and treat it as untouchable. This is called deadline propagation, and gRPC builds it in: a deadline set by the caller travels with every downstream call, so each service knows how much time it really has. You can copy the idea in a dozen lines. Pass the deadline object into your tool wrapper, and have the wrapper refuse to start a call when the remaining time is below that tool's p50. Starting a call you already know can't finish is the wasteful case. The model calls count against this clock as well. A slow generation at step three shrinks what's left for step four, and your tool wrapper should know that without being told. Agent tool timeout and retry budget Retries are where good timeouts go to die. Say your HTTP client retries twice, your tool wrapper retries twice, and your agent loop lets the model retry the tool up to three times. That's not three attempts, it's up to eighteen, each one waiting its full timeout. The Builders' Library piece warns about exactly this multiplication across layers. Pick one layer to own retries. Usually that's the tool wrapper, with the SDK retries switched off and the model told it shouldn't repeat a failed call more than once. Then bound the retries with the deadline, not just a count. A retry is only allowed if the remaining time exceeds the tool's timeout plus the reserve. Add jitter to the delay between attempts, a random spread of a few hundred milliseconds, so that many agents hitting the same recovering API don't all come back at once. The harder question is which calls are safe to repeat at all. A timeout doesn't mean the request failed. It means you stopped listening. If a tool creates an invoice, sends an email or charges a card, the first attempt may have succeeded on the server while your client gave up waiting. Retrying blindly gives you two invoices. For writes, use an idempotency key, which Stripe's API supports through its Idempotency-Key header, so a repeated request returns the original result rather than doing the work twice. If the API has no such mechanism, don't auto-retry the write. Surface the uncertainty instead and let a person or a read-only check confirm what happened. Reads are different. A search, a lookup or a status check can be repeated freely, and those are the calls where one retry with a slightly longer timeout often rescues the run. What the agent should do when a tool call times out This is the part most guides skip, and it decides whether the run recovers or spirals. When the wrapper cuts a call off, don't raise an exception that kills the run, and don't return an empty string. Hand the model a structured error as the tool result, the same way you'd hand it any other result. Anthropic's tool use format has an is error flag on tool results for exactly this, and other APIs accept an ordinary error message in the same slot. Make the error useful. Include the tool name, how many seconds it ran, whether the call is safe to retry, and what's still left in the run budget. Something like: the order lookup timed out after 3 seconds, a retry is allowed once, 41 seconds remain. A model given that can reason. A model given a bare error message tends to try the identical call again, and again. Then give it a short policy in the system prompt, with three outcomes. Retry once if the error says retrying is allowed. Switch to another tool if one covers the same need, say a cached copy or a second data source. Otherwise stop and tell the user plainly which step failed and what's already done. A partial answer with a clear note, such as the first four of five pages summarized and the fifth source not responding, beats a run that never ends, and users forgive it far more readily than a spinner. And cancel what you abandoned. When the timeout fires, the underlying request is often still running. In asyncio, cancel the task, and in a thread pool, make sure the client actually closes the socket. A zombie call keeps holding a connection, and if the tool is an expensive one on the other end, you may still be billed for work whose result you threw away. Check it works before it matters You can test all of this without waiting for a real outage. Stand up a fake tool that sleeps for a configurable time, point your agent at it, and run three cases: a response just under the limit, a response just over it, and a response that never comes. In the first the run should complete untouched. In the second, you should see one retry or one clean fallback. In the third, the run should finish near the deadline with an honest partial answer, not at the SDK's ten minute default. After launch, treat timeouts as a metric. Log every timeout with the tool name, the limit that fired, and whether it was the per-call or the run deadline. Watch the timeout rate per tool. If it sits well above the false-timeout rate you chose, say one percent for a p99 setting, the tool has slowed down or your number was too tight. If it sits at zero for a month, you may be too generous, and a tighter limit will protect you better. The distribution drifts as vendors change their infrastructure, so rerun the percentile query monthly. A tool that was fine at a 3 second timeout in October can have a p95 of 5 seconds by January. Do this in order. Add timing to the tool wrapper this week, set per-tool limits from the first full week of data, then add the run deadline and the structured timeout error. The first step alone will show you which of your tools has been quietly eating your runs. Also read: How to Build a Golden Dataset for AI Agent Evaluation From Your Logs https://startupfortune.com/how-to-build-a-golden-dataset-for-ai-agent-evaluation-from-your-logs/ • How to write an AI agent memory retention policy before it bites https://startupfortune.com/how-to-write-an-ai-agent-memory-retention-policy-before-it-bites/ • How to calibrate LLM as a judge before your eval dashboard lies to you https://startupfortune.com/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you/ This article is posted in AI News https://startupfortune.com/category/ai/ , check it out for more related stories. Join the discussion Open in the community → https://startupfortune.com/community/ Almost there. Sign in and your reply posts straight away.