AI Agent Tool Timeout: What a 90-Run Deadline Test Found

A 90-run local Python asyncio test by AgentSEO cofounder Daniel Martin found that resetting a 40-millisecond timeout at each of three sequential tool calls let all 30 runs report success, with 21 finishing more than 45 milliseconds after start, while a shared end-to-end deadline returned a timeout in 27 of 30 runs and never reported a late success. A deliberately broken variant that swallowed cancellation returned ok in all 30 runs, 27 of them late. The measurements came from sequential asyncio.sleep calls on one macOS machine on 9 November 2026 using Python 3.12.1, not from AgentSEO, a hosted model, an MCP server or real user traffic, and Martin noted the per-call pattern can log three green tool calls while the user sees a missed latency budget.

An agent can finish every tool call “within timeout” and still keep the user waiting longer than the promised budget. A small version of that failure was tested with 90 local Python asyncio runs. The interesting part was not a slow tool. It was where the clock restarted. AI assisted with the fixture, charts, and first draft. Daniel Martin is an AgentSEO cofounder. The measurements are from sequential asyncio.sleep calls on one local machine, not from AgentSEO, a hosted AI model, an MCP server, or real user traffic. That distinction matters throughout. Suppose an agent performs three dependent steps: fetch context, run a search, then enrich a result. Each step is allowed 40 milliseconds. If the calls take 15 milliseconds each, all three pass their local checks. The whole action takes about 45 milliseconds. A 40-millisecond promise to the user has already been missed. This difference shows up in real agent APIs because a tool timeout is often documented separately from a run or workflow timeout. The OpenAI Agents SDK tool documentation https://openai.github.io/openai-agents-python/tools/ describes per-call timeouts for asynchronous function tools. The AG2 tool documentation https://docs.ag2.ai/docs/user-guide/tools/tools/ likewise discusses tool-level timeouts and cancellation. Those documents describe their own implementations; this article tests only a generic Python fixture, not either SDK. The test asks two narrower questions. First, how often does resetting the same timeout at each sequential call let a locally constructed chain report success after the end-to-end budget? Second, what happens when a tool catches cancellation and continues? The runnable fixture uses Python 3.12’s asyncio.wait for and asyncio.timeout . It defines ten three-step latency profiles, listed in the saved summary.json and research.py. Their nominal totals range from 24 to 60 milliseconds. Each profile runs three times under three policies, giving 10 × 3 × 3 = 90 local runs. The policies were: The third policy is intentionally wrong. It tests a failure mode, not advice to suppress cancellation. Python’s asyncio documentation https://docs.python.org/3.12/library/asyncio-task.html task-cancellation says cancellation is raised at a suspension point, cleanup belongs in try/finally, and a coroutine that catches cancellation should generally propagate it after cleanup. It also warns that asyncio.timeout and structured concurrency can misbehave if cancellation is swallowed. Wall time was measured with time.perf counter . The fixture recorded status and completed steps and saved every row to results.csv. “Late success” here means the case returned ok more than 45 milliseconds after start: the 40-millisecond budget plus a five-millisecond margin to keep small scheduler noise out of the headline. The results are from one macOS run on 9 November 2026, using Python 3.12.1. Re-running may move cases near the boundary. Results across 30 runs per policy: The per-call reset finished all 30 runs with ok. Twenty-one were late by the test's conservative rule. None of its individual sleeps was close to 40 milliseconds, so a per-call check never exposed the overall miss. This is the trap: the code can log three green tool calls while the user sees a red end-to-end latency number. The shared deadline returned a timeout in 27 of 30 runs and never reported a late ok. Only 21 of the 30 profiles had a nominal sum above 40 milliseconds. The additional timeout cases sat on or close to the boundary; event-loop scheduling and wrapper overhead explain why a nominal 40-millisecond chain cannot be assumed to finish in 40 wall-clock milliseconds. The result is a property of this tiny local setup, not a prediction of a production timeout rate. The cancellation-swallowing version returned ok for all 30 runs, with 27 late. It had a shared timeout on paper, yet the deliberately broken step consumed the cancellation, slept 15 milliseconds more, and let the chain complete. A deadline is only as reliable as the code beneath it. In a networked agent, cancellation may also have to propagate through an HTTP request or remote task; this fixture did not test that layer. Here is a copyable version of the whole comparison. It runs all ten profiles three times under each policy and prints late ok counts, timeout counts, and median elapsed time. Run it with Python 3.12 or newer. Cases near 40 or 45 milliseconds can change with scheduler load; compare the pattern, not the exact wall-time decimals. python import asyncioimport statisticsimport timebudget = 0.040profiles = 8, 8, 8 , 10, 12, 15 , 15, 15, 15 , 20, 20, 20 , 5, 20, 20 , 25, 8, 8 , 8, 25, 8 , 8, 8, 25 , 30, 5, 5 , 12, 18, 18 , policies = "per call", "shared", "swallow cancel" async def tool ms, swallow=False : try: await asyncio.sleep ms / 1000 except asyncio.CancelledError: if not swallow: raise await asyncio.sleep 0.015 Deliberately broken cancellation.async def trial delays, policy : start = time.perf counter outcome = "ok" try: if policy == "per call": for ms in delays: await asyncio.wait for tool ms , budget else: async with asyncio.timeout budget : for ms in delays: await tool ms, swallow=policy == "swallow cancel" except TimeoutError: outcome = "timeout" return outcome, time.perf counter - start 1000async def main : rows = {p: for p in policies} for in range 3 : for delays in profiles: for policy in policies: rows policy .append await trial delays, policy for policy, runs in rows.items : late ok = sum status == "ok" and ms 45 for status, ms in runs timed out = sum status == "timeout" for status, in runs median ms = round statistics.median ms for , ms in runs , 2 print policy, len runs , late ok, timed out, median ms asyncio.run main On a fresh rerun, the per-call late count moved from 21 to 22, while the shared-deadline late count stayed at zero and the broken cancellation case stayed at 27. That variation is why 21/30 should not be treated as an estimated production failure rate. It is the outcome of the archived sample. The mechanism is the point: a timeout value means little until you say what span of work it covers. Carry one absolute deadline from the user request through each nested step. Before a new tool call, calculate remaining time from the event loop clock. If the remaining budget is gone, stop before starting more work. A smaller per-tool cap can still protect one unusually slow call, but it should be bounded by the remaining end-to-end time. For Python coroutines, keep cleanup in finally and re-raise CancelledError after any explicit catch. Test a callee that ignores cancellation on purpose. The pass condition is not just that a timeout exception appeared somewhere in logs. It is that no late success was returned and no abandoned work continued after the caller stopped waiting. Record three timestamps in traces: action start, each tool start and end, and action end. Then report both the tool’s timeout status and the whole action’s elapsed time. A green per-tool status alone cannot prove the user’s deadline was kept. For tools with external side effects, add an operation ID and a separate completion check before retrying. Timeout can mean “the caller stopped waiting,” not “the work definitely did not happen.” This fixture is deliberately small. It uses sequential local sleeps and one scheduler, without retries, parallel fan-out, model latency, queueing, transport cancellation, or side effects. Its value is as a regression test for a specific boundary: does the agent preserve the caller’s deadline, and do tools honor cancellation? Run the same idea with your actual SDK and network tools before treating a timeout setting as a service-level guarantee. AI Agent Tool Timeout: What a 90-Run Deadline Test Found https://pub.towardsai.net/ai-agent-tool-timeout-what-a-90-run-deadline-test-found-601d84997b31 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.