Disclosure: I'm Nabeel, founder of VoiceDash, a client portal for Retell agencies. This is an adapted republish of a guide from the VoiceDash blog, rewritten for developers. Nothing below needs VoiceDash; it uses Retell's own features plus a little glue code.
TL;DR: A voice agent that passed every test on launch day can quietly break a month later: a calendar token expires, a webhook starts failing, the business changes its hours, or a provider has a bad afternoon. The agent keeps answering and sounds fine, and books nothing. Monitoring in production means watching four layers (is the line answering, are tools working, is quality holding, is usage normal), wiring a failover before you need it, and routing alerts to one place. Retell now ships most of the primitives; this post maps each layer to the field or feature that covers it.
Pre-launch testing proves the agent worked on the day you shipped it (my checklist for that is in How to Test a Voice AI Agent Before Go-Live). Production is different, because the world around the agent keeps moving:
None of these stop the phone ringing. That is what makes them dangerous.
The loud failure: calls ring out, drop on connect, or end in an error. Rare, and the one clients never forgive.
Heartbeat call. Once or a few times a day, place an automated test call to each production number from a separate number. Check that it connects, the agent greets within a normal window, and the call ends cleanly. A small scheduled workflow (n8n, a cron job, anything) plus a Twilio number does it. Alert after two consecutive failures, not one.
Volume floor. Every client has a rhythm. Compare today's call count against the same weekday's average and alert when it falls far below during business hours. Zero calls by noon at a busy office is usually a forwarding change on the client side, not an AI problem, but it is still yours to catch.
Disconnection reasons. Every Retell call records a disconnection_reason, available in Call History and on the Get Call API. Retell's disconnection table splits them roughly into three buckets:
expected (call_status = ended)
user_hangup, agent_hangup, call_transfer, voicemail_reached,
inactivity, max_duration_reached, user_requested_dnc ...
not connected (outbound)
dial_busy, dial_no_answer, dial_failed, invalid_destination,
marked_as_spam, network_blocked ...
errors (call_status = error)
concurrency_limit_reached, error_llm_websocket_*, error_asr,
error_no_audio_received, error_retell, error_unknown ...
Log the field for every call and alert on any spike in the error bucket. Two details worth knowing from the docs: dial_failed is also what you see when carriers have flagged your outbound number as spam after many short calls, and the error_llm_websocket_* family only appears when you run a custom LLM over Retell's websocket.
The quiet failure, and in my experience the most common one. The conversation sounds perfect; the booking, CRM update or follow-up text never happens.
Log every tool call and its result. Whether the agent calls your endpoint directly or goes through a workflow tool, record the request, response, status and latency, keyed by call_id. If tool results are not logged you cannot see this failure at all.
Give every workflow an error branch. Every workflow the agent depends on needs an error path that alerts with the client name, the workflow and the failing step.
Make the agent honest about failure. The prompt should say exactly what to do when a tool returns an error: apologise, take the caller's details, promise a callback, flag the call. It must never confirm a booking that did not go through. A caller who gets a callback is fine; a caller who shows up for an appointment that does not exist is a complaint.
Reconcile against the source of truth. Once a day, compare calls marked booked against bookings that actually landed in the client's calendar or CRM:
// nightly reconcile, per client
const booked = calls.filter(c => c.call_analysis?.custom_analysis_data?.outcome === "booked");
const landed = await calendar.listEventsCreatedBetween(start, end, { source: "voice-agent" });
if (booked.length !== landed.length) {
alert(`${client.name}: ${booked.length} calls say booked, ${landed.length} bookings exist`);
}
The outcome field here is whatever you defined in post-call analysis; the calendar call is your own integration.
Some failures are not errors. The agent answers, the tools work, and it still handles callers badly because the business changed or an edge case became common.
Post-call analysis as the quality signal. Extract a structured outcome for every call (booked, answered question, transferred, message taken, spam) plus a flag for a frustrated caller or a request for a human. Track the rates per client week over week. A jump in transfers or frustrated callers means the prompt no longer matches reality.
A weekly transcript sample. Numbers tell you something changed; transcripts tell you what. Skim a handful of calls per client each week, weighted toward transfers, very short calls and anything flagged frustrated.
Latency, per component. After a call ends, Get Call returns a latency object with p50, p90, p95, p99, min, max for components such as e2e, llm, tts, asr and knowledge_base (not every field is present on every call). Store e2e.p90 per call and watch its trend. If it creeps up, the breakdown tells you whether it is the model, the voice or retrieval. Note that e2e excludes turns where the agent only ran a tool without speaking, so slow tools need their own timing from your logs.
Usage anomalies are both a cost problem and a symptom:
Set a maximum call duration on every agent (it ends calls with max_duration_reached), then alert when a client's daily minutes run well above normal.
Retell has alert rules that evaluate aggregate metrics on a schedule and send an email or a webhook when a threshold is crossed. Metrics include call_count, call_success_rate, negative_sentiment_rate, custom_function_latency, custom_function_failure_count, transfer_call_failure_count, concurrency_used, total_call_cost and api_error_count. Rules can be absolute ("success rate below 90% over the last hour") or relative to the previous cycle, and can be filtered by agent, environment tag, disconnection reason or post-call extraction field.
Two constraints shape how you use them:
Monitoring tells you something broke; failover decides how much it costs.
The fastest way to kill a monitoring setup is noise. If every short call pings your phone, you will mute the channel within a week and miss the one that mattered.
The agent that worked on launch day is not the agent that is answering today, because everything around it changed. Watch the line, the tools, the quality and the usage; put disconnection_reason, latency and your post-call outcome on every call record; let Retell's alert rules cover the aggregates; and set the fallback number before the first incident, not during it.
Drafted with AI assistance and fact-checked against each vendor's docs.