{"slug": "how-to-measure-whether-your-ai-receptionist-is-working", "title": "How to measure whether your AI receptionist is working", "summary": "A developer who built a Blue Ridge missed-call demo on Retell and GPT-4.1 outlined a five-metric method for judging whether an AI phone receptionist actually works: calls answered, calls with a logged outcome, hand-offs, bookings, and wrong answers. The approach requires the agent to call a log_outcome function before hanging up rather than trusting a platform's \"call successful\" flag, which the developer called \"a model's opinion of a transcript, not a count of jobs.\" The developer also reported a simulation suite of 38 callers and 114 calls per run at roughly $0.25, whose first run caught 5 missed emergency handoffs in 41 plumbing calls on a cheaper model.", "body_md": "Track five numbers: how many calls it answered, how many calls ended with a logged outcome, how many it handed to a person, how many turned into bookings, and how many answers were wrong. The first four come from your call log. The fifth only comes from a person reading calls.\n\nDo not rely on the platform's \"call successful\" flag alone. It is a model's opinion of a transcript, not a count of jobs.\n\nAn agent that picks up every call can still fail every caller. It can answer, chat, quote a price you never set, and hang up with no booking.\n\nSo the first number is only a floor. Pull it from your call log, then ask what happened on each call.\n\nA made-up plumbing example: 60 calls in a week, all answered. That tells you the phone worked. It does not tell you whether 12 or 2 of those callers booked a visit.\n\nMost voice platforms do part of this work for you. Retell's [post-call analysis](https://docs.retellai.com/features/post-call-analysis-overview) writes results onto each call after it ends. The default fields are a summary, the caller's sentiment, whether the call met your success criteria, and whether the agent reached a voicemail. You can add your own fields, such as a yes/no \"booked\" or a pick-one \"reason for call.\"\n\nVapi's [call analysis](https://docs.vapi.ai/assistants/call-analysis) offers the same idea: a summary, structured data pulled out by a JSON schema, and a success evaluation with scoring rubrics.\n\nTwo limits matter. Retell notes that sentiment comes from the transcript and \"does not use tone or pitch,\" so an angry voice reading polite words can look fine. And, per the same page, custom fields are not filled in for calls that were not connected.\n\nTiming matters too. Retell's [webhooks](https://docs.retellai.com/features/webhook-overview) send `call_ended` when a call completes, and `call_analyzed` later, once analysis is done. Only the second carries the analysis. If you build a report, listen for the second one.\n\nDon't ask a model to judge from the transcript whether the call went well. Make the agent record what happened.\n\nIn my Blue Ridge missed-call demo, a single-prompt Retell agent on GPT-4.1, the agent must call a `log_outcome` function before it hangs up. The lead then moves through a pipeline: Missed, Texted, Called, Booked. Every stage is a count you can read, and a call with no logged outcome stands out.\n\nThat is a prompt rule, so it can fail. Count it. If 3 calls out of 40 have no outcome (made-up numbers), you have found a gap in the prompt before a customer finds it.\n\nYou can [hear that agent on the recovery demo](https://voicedesk-h31g.onrender.com/agent/recovery). In that demo the text-back goes out in 8 seconds and the AI callback connects in 52.\n\nI won't give you a benchmark, because I have no honest one to give. A plumber's emergency line and a dental front desk should look very different. Set your own baseline in week one, then watch for change.\n\nHere is a made-up example, labeled as made up. In week one, 50 calls: 48 answered, 44 with a logged outcome, 9 hand-offs, 14 bookings. That is a baseline. If next week shows 30 with a logged outcome, something broke, even if answered calls stayed high.\n\nRead the hand-offs more than the bookings. A hand-off means the agent decided it could not help. Your [emergency policy](https://adelatya.com/blog/ai-phone-agent-emergency-calls/) and your [fallback for unknown questions](https://adelatya.com/blog/ai-receptionist-doesnt-know-the-answer/) both show up there.\n\nYou cannot count them from a dashboard. A success flag is a model's opinion of the transcript, so a confident wrong price can slip through.\n\nDo this instead:\n\nBefore launch, I test with simulated callers. My suite runs 38 simulated callers, 114 calls per run, for about $0.25 a run. Its first run caught 5 missed emergency handoffs in 41 plumbing calls on a cheaper model. The agents in that suite run on GPT-4o-mini, with a GPT-4.1-mini judge.\n\nOpenAI's GPT-4.1 [prompting guide](https://developers.openai.com/cookbook/examples/gpt4-1_prompting_guide) says models are \"inherently nondeterministic\" and recommends \"building informative evals and iterating often.\" The same applies to a phone agent: the same prompt can behave differently on a different call, so you measure it over many calls. My post on [how we test phone agents](https://adelatya.com/blog/how-we-test-ai-phone-agents/) explains the method.\n\nAn AI receptionist is working if it beats what you had before. So measure the old way first.\n\nBefore you switch it on, count for one week how many calls go unanswered and how many of those callers ever call back. That is your before number. Without it, a 14-booking week means nothing.\n\nThen compare cost. My voice agent build is $900, then $450 a month with 500 minutes, and 15 cents a minute after that ([pricing](https://adelatya.com/pricing/)). Made-up break-even: if each extra job earns you $225 in profit, two extra jobs a month cover the $450 fee, and the $900 build takes a few more months to repay. If it books none, you have your answer.\n\nIf you get five calls a week, a dashboard is overkill. Listen to every call and skip the metrics.\n\nAlso, if you can't tell a real booking from a fake one, because your calendar is a mess or staff change bookings by hand, fix that first. No agent can look good against numbers you can't trust. In that case a human answering service may be the better choice.\n\nIf you want these fields set up on a [voice agent build](https://adelatya.com/voice-agent-engineer/), ask for the outcome logging first.\n\n*Originally published at [adelatya.com](https://adelatya.com/blog/measure-ai-receptionist-working/).*", "url": "https://wpnews.pro/news/how-to-measure-whether-your-ai-receptionist-is-working", "canonical_source": "https://dev.to/adelatya/how-to-measure-whether-your-ai-receptionist-is-working-1f7o", "published_at": "2026-11-22 06:37:42+00:00", "updated_at": "2026-11-22 06:42:27.348577+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools"], "entities": ["Retell", "Vapi", "OpenAI", "GPT-4.1", "GPT-4o-mini", "GPT-4.1-mini", "Blue Ridge"], "also_reported_by": [], "takeaway": "Have the voice agent call a log_outcome function before hanging up and count calls with no logged outcome, since a platform's success flag is a model's opinion of the transcript rather than a record of what happened.", "recommendation": {"advice": "Log a structured outcome on every call and count the gaps, then run simulated callers before launch to catch missed hand-offs a success flag would hide.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/how-to-measure-whether-your-ai-receptionist-is-working", "markdown": "https://wpnews.pro/news/how-to-measure-whether-your-ai-receptionist-is-working.md", "text": "https://wpnews.pro/news/how-to-measure-whether-your-ai-receptionist-is-working.txt", "jsonld": "https://wpnews.pro/news/how-to-measure-whether-your-ai-receptionist-is-working.jsonld"}}