Every Tool in My Voice Agents Had a Two-Minute Timeout. Callers Give You Eight Seconds.

A developer who builds voice agents for clients audited nine Retell agent exports and found every custom tool set to a 120-second timeout with speak_during_execution disabled, meaning a stalled webhook can leave callers in silence for up to two minutes before the agent's retry-and-apologize rule even triggers. The developer argues timeouts should sit near the real p99 of the backing workflow plus headroom, retries should be dropped for non-idempotent actions like bookings, and every phone number should fall back to a human, ring group, or checked voicemail so an AI outage never produces a dead line.

When a web service goes down, the user gets an error page. Ugly, but honest. When a voice agent goes down, the caller gets silence, and silence on a phone line reads as "this business is closed." They hang up and call the next result on Google. Nobody files a bug report. So I now design the failures first. This is the engineering version of an outage plan I wrote up for clients, plus a confession from my own agent exports. When I scope an agent, I write out every service a single call touches, in order: "The AI is down" is not one incident. It is five or six, and each one sounds different to the caller: The design job is deciding what each failure sounds like. My first rule is that the phone number must always end at something that works without the AI: a human, a ring group, or at worst a voicemail box someone actually checks. If the number lives in Twilio, configure its fallback handler, which runs when the primary one fails, to forward to a backup line or record a voicemail and email it. The worst outage becomes a slightly worse phone experience instead of a dead line. If the business forwards its line to the agent, overflow forwarding staff ring first has a fallback built in; all-calls forwarding has none unless you build one. Either way, the client can turn forwarding off in two minutes. That is the manual circuit breaker, and it does not need me awake. The test I use: if the voice platform vanished right now, what would a caller hear? If the answer is "nothing," fix that before touching the prompt. This is where my own exports got interesting. I keep JSON exports of nine Retell agents I have built in one folder I have mined it before for tool parameter design https://dev.to/nabeelbaghoor/every-tool-parameter-is-a-question-your-voice-agent-has-to-ask-5512 and a config diff that went wrong https://dev.to/nabeelbaghoor/my-hvac-voice-agent-was-listening-for-the-word-invisalign-273g . I grepped them for failure handling, expecting to feel good. The good news: every single prompt has a tool failure rule. The shape is identical across all nine: If a tool fails: apologize briefly, retry once. If second failure: "I apologize for the inconvenience. Let me have someone from the team call you back to get this scheduled." End call. Right instinct: no invented slots, no infinite loop, a human promise at the end. Reading it as an engineer, though, four things bother me. Every custom tool across the nine exports has the same setting: { "type": "custom", "name": "checkCalendar", "timeout ms": 120000, "speak during execution": false } So "if a tool fails" can mean the model waits up to two minutes before it learns anything failed, and "retry once" can double that. Meanwhile speak during execution is false, so nothing is said while the request hangs. The prompt does tell the agent to say a short bridge like "let me check on that" before calling a tool, but a bridge covers a second or two, not a stalled webhook. A two-minute timeout on a user-facing web request would get flagged in review. On a phone call, a caller gives you maybe eight seconds of silence. I had tuned the prompt rule and never revisited the timeout under it. It should sit near the real p99 of the n8n workflow plus headroom, and the retry should not exist for anything that is not idempotent. Booking twice because the first request was slow, not failed, is its own bug; I wrote about that class of problem in you can retry a write, you cannot retry an email https://dev.to/nabeelbaghoor/you-can-retry-a-write-you-cannot-retry-an-email-lfo . Several of the fallback lines tell the caller to call the business back at its main number. That is fine if that number reaches staff. If that number is the one forwarded to the agent, the fallback is a loop: the caller rings back into the same broken tool and hears the same apology. I have not verified the routing for every one of those agents, and that is the point: the fallback text lives in the prompt, the routing lives in a carrier dashboard, and nothing connects them. The fix is a different path, not a different sentence: a transfer that bypasses the agent, or a captured message and a callback. The agent has usually collected a name, a number and the reason for calling by the time the calendar fails. Then it ends the call. Whether those details survive depends on whether the post-call pipeline captures them, and if the failing tool is the CRM write, writing the message to the CRM does not help. My rule now is that caller details land somewhere independent of the failing tool: a team notification or a stored record that a post-call workflow can replay once the CRM is back. Only 2 of the 9 exports have a transfer call tool. For the rest, a tool outage leaves the agent with exactly two moves: apologize, or hang up. A tool outage is precisely when a live transfer earns its keep, which is the argument I made in your voice agent needs three exits, not one https://dev.to/nabeelbaghoor/your-voice-agent-needs-three-exits-not-one-473h . My own exports are the counterexample. Every export has fallback voice ids set to two alternate voice providers, so a bad day at the primary text-to-speech provider means a different voice, not a mute agent. Real automatic failover, and it works because the platform owns both sides of the switch. I did not design it, which is also why it was the one layer already covered. The embarrassing outage is the one the client reports to you at 10am. Two checks catch most of what dashboards miss. A scheduled synthetic call. A real call to the agent's number from outside the stack: wait for the greeting, ask one scripted question, check the post-call event arrived. It is the only thing that catches top-of-chain failures, because a broken forwarding rule produces no error in the voice platform. From its point of view, no call ever arrived. Zero-volume alerts. Errors are easy to alert on. Silence is not. Track normal call volume per hour per business and alert when it drops to zero during hours that usually have calls. A clinic that gets a dozen calls on a weekday morning and has received none since 8am has a problem, even if nothing threw. Automatic switching between, say, two LLM providers can fail in more confusing ways than the outage itself. An untested model behind the same prompt is a different agent, and it needs the same regression tests https://dev.to/nabeelbaghoor/you-cannot-unit-test-a-summary-you-can-still-test-it-i7d as any other change. So each agent gets a one-page runbook written with the client: who gets alerted, the kill switch with screenshots of the carrier or Twilio setting , and who works through the message backlog afterward. The honest target is not an agent that never goes down. It is that no outage produces dead air, loses a caller's details, or lasts longer than it takes a named human to notice and flip a switch. The full client-facing version of this outage plan is on my site: What Happens When Your AI Voice Agent Goes Down https://nabeelbaghoor.com/blog/voice-agent-outages-failover/ . What does your stack do when a dependency stalls instead of failing? I am curious whether anyone has found a clean way to keep prompt-level fallback text and carrier-level routing in sync, because right now mine live in two places that never talk.