{"slug": "ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent", "title": "AI Called Me Back — Recoverflow Dev Diary Day 2: Two Hours with the Voice Agent", "summary": "A developer building Recoverflow's voice agent for collections calls found that a one-way script would fail because it couldn't respond to customer replies. Using ElevenLabs' Conversational AI, the developer created a two-way voice agent with 18 dynamic variables to personalize each call, selecting the 'Sarah' voice for its calm, professional tone suited to sensitive debt collection conversations.", "body_md": "After Day 1 wrapped, I was waiting for Day 2 — Voice Agent day.\n\nHonestly, Voice Agent was the piece I was most nervous about in this whole system.\n\nBecause the other agents — write emails, look up data, run routing — at worst they write something mediocre, get blocked by Tone Coach, and rewrite once. But Voice Agent actually picks up the phone and calls a real person. The other side picks up, hears Sarah's voice, hears the script I wrote, gets the routing I designed.\n\nAny mistake in there, the person who hangs up doesn't tell you \"your AI was weird.\" They just remember \"that company was unpleasant.\"\n\nCollections is especially sensitive on this point because the other side is already in a bad mood, right? They might have real cash flow problems, might have fought with their own customer, might be waiting on someone else to pay them. Your AI steps on one more landmine and that's the last straw that nukes the relationship.\n\nSo all day on Day 2 I was thinking about two things:\n\n**First, Voice Agent has to be two-way, not one-way.**\n\n**Second, if it's two-way, it has to understand what humans actually say.**\n\nThe original Voice Agent design was — we call out, talk, hang up. Like an audio version of a reminder email.\n\nBut when I was writing the Day 55 last-chance call section, I read the script out loud once, and it felt off.\n\nThe script had Sarah say:\n\n\"I know things come up. We can work out a payment plan if you need.\"\n\nBut — what's supposed to happen after she says that?\n\nIf the customer responds \"yeah, next week,\" \"I need to discuss with my boss,\" \"actually we have cash flow problems right now\" — a one-way voice **can't hear that**, right?\n\nSo this line \"we can work out a payment plan\" is just talk, then we hang up the next second?\n\nThis design contradiction was already baked into decision D-027. **The design intent was conversation, so voice had to be two-way.** Day 2's goal was to finish making it that way.\n\nWhen I registered for ElevenLabs on Day -1 (6/11), I chose ElevenAgents — their Conversational AI product.\n\nAt the time J told me \"first use the free tier 15 min to run dev playground, doesn't burn actual minutes, upgrade to $6 Starter when you actually want to dial out.\"\n\nSo I did that.\n\nDay 2 the first thing I did was go to the ElevenLabs platform and build out the whole ConvAI Agent.\n\nThe voice I picked was Sarah — voice ID `EXAVITQu4vr4xnSDxMaL`\n\n. This is a premade voice that ships with ElevenLabs, free, positioned as \"mature, reassuring, confident female US-EN.\"\n\nI listened to a few voice samples. The reason I picked Sarah was simple: **she sounds like the voice I'd want to pick up the phone to**.\n\nNot too young, not over-sweet, not a cold customer-service-robot voice. It's that... you'd call into a mid-sized US bank's customer service line and might end up talking to some middle-aged senior advisor — calm, warm, but with professional distance.\n\nCollections needs exactly that voice.\n\nToo young and the customer doubts your company's professionalism, too old-school and it feels heavy-handed. Sarah sits right in the middle...\n\nThe ConvAI Agent's prompt is fixed, but every call is a different customer — different company name, different outstanding balance, different days past due, different patterns.\n\nTo make Sarah sound like she's actually talking to a specific customer, the case context has to be injected before the call.\n\nElevenLabs does this with dynamic variables — you write `{{customer_first_name}}`\n\n, `{{invoice_amount}}`\n\n, `{{days_past_due}}`\n\nin the prompt, and during the API call before dialing you pass those variables in. Sarah will then naturally speak the corresponding values.\n\nWe ended up with 18 variables.\n\nCustomer last name, company name, outstanding balance, days past due, pattern_tag (this customer's past payment-stalling pattern), recent_excuse (last excuse they used)...\n\nEvery line Sarah speaks carries this context. She won't say \"Hi this is Sarah calling about your invoice\" — she'll say \"Hi, this is Sarah calling for ABC Trading regarding your invoice from March.\"\n\nSpecific enough that the other side thinks \"wait, this doesn't sound like a spam call.\"\n\nSarah doesn't just talk. She has to do four things in the call:\n\nFirst, when the customer states an excuse (cash flow issue / dispute / overseas wire delayed), she logs the category — `log_excuse(category)`\n\n.\n\nSecond, when the customer commits to a payment date, she records the date and amount — `propose_payment_date(date, amount)`\n\n.\n\nThird, when the other side says \"I need to talk to a human,\" she requests a callback — `request_callback_to_human()`\n\n.\n\nFourth, if the situation needs escalation (other side is hostile, mentions bankruptcy, mentions a lawyer, was identified as the wrong person), she throws it back to a human — `escalate_to_concierge(reason)`\n\n.\n\nThese 4 tool callbacks are Sarah's \"hands and feet.\" Without them she just talks; with them she actually does things in the conversation.\n\nThe way you connect those tools is — in the ElevenLabs agent setting, you point each tool to a webhook URL. When Sarah is mid-conversation and decides it's time to call this tool, she sends a webhook to our server. Our server receives it, writes to the audit trail, returns the result, she continues the conversation.\n\nAfter wiring all of this up I was very excited — because this is what a real \"can listen, can do\" voice looks like.\n\nBut excitement aside, I don't trust anything I haven't tested.\n\nThis next part is the most dramatic part of Day 2...\n\nI told J: \"I want to call myself once and see.\"\n\nThe Twilio trial has a constraint that you can only call verified numbers, so I verified my own Taiwan phone +886-XXX-XXX-XXXX first. I was in Korea at the time, so this call would go cross-border — from a Twilio US number to a Taiwan number, with ElevenLabs's Sarah speaking English to me (pretending to be ABC Trading's boss).\n\nWhen the phone rang I was so excited — wondering what it actually feels like when an AI calls you, whether it'd sound like a real person...\n\nI picked up. The Twilio trial's \"press any key to continue\" disclaimer played first (this part gets edited out of the demo video for free-tier accounts). Then — Sarah's voice came in.\n\n\"Hi, this is Sarah calling for ABC Trading. I'm reaching out regarding the outstanding invoice...\"\n\nThe voice was genuinely natural. No AI-synth artifact, no weird pacing, the tone was professional but warm.\n\nI took a breath, started playing the role — pretending to be ABC Trading's boss, I replied:\n\n\"I need to ask my boss about this, I can email you back in 2 days.\"\n\nThis is a very realistic response, right? — in B2B, the person who picks up isn't usually the final decision-maker, they need to discuss internally, wait for approval. Totally normal cooperative behavior.\n\nThen...\n\nSarah did NOT treat what I just said as cooperative.\n\nOn the other end of the phone she said something I will never forget:\n\n\"I understand you need time to consult internally. However, we've already made multiple attempts to resolve this matter. At this point, I'll need to escalate this to our recovery team for further action.\"\n\nWait.\n\n**Wait wait wait wait.**\n\nShe treated my \"reply in 2 days\" as evasion.\n\nAs \"this customer is stalling again.\"\n\nAs \"time to escalate to the next stage.\"\n\nI just stood there frozen. Then immediately hung up and ran back to the computer.\n\n\"That's not right...\" I said to J, \"reply in 2 days isn't stalling, it's cooperation. I'm working with you, I just have to go through an internal process.\"\n\nJ's response was instant: \"Right. Phase 3b needs to be added.\"\n\nPhase 3b is a new sub-branch in the Voice Agent prompt, grown from this one bug-bitten call.\n\nThe scenario it handles is very specific: **the customer is committing to a follow-up DATE, not a payment date**.\n\nThe original prompt structure was:\n\nThe problem with the original \"3\" was it was too binary. Either pay or evade. But the real world has a third choice — **\"I need to ask someone then get back to you\"**.\n\nThis isn't payment, and it isn't evasion. It's the middle state.\n\nThe new Phase 3b sub-branch reads:\n\n\"IF customer commits to follow-up RESPONSE date (not payment date) → accept gracefully\"\n\nTool:`log_excuse(category=\"approval_workflow\")`\n\nConfirm: \"So you'll email Judy by [date], correct?\"\n\nTool:`escalate_to_concierge(reason=\"customer_committed_to_followup\")`\n\n(non-negative escalation — ball is back in the human's court)\n\nThe key is that last `escalate_to_concierge`\n\n— the reason isn't \"customer_evading,\" it's \"customer_committed_to_followup.\"\n\nSame escalation primitive, completely different meaning.\n\nThe former means \"this customer has a problem, prepare to escalate to the next stage.\"\n\nThe latter means \"the ball is in the customer's court, we wait for their follow-up, this call is done.\"\n\nThe two reasons hit the Concierge layer and trigger completely different downstream flows.\n\nAfter Phase 3b was added, I ran another simulation — not a real call this time, but ElevenLabs's simulate-conversation API (which doesn't burn actual voice minutes — pure text validation of the dialogue script).\n\nThis time I input: \"I need to ask my boss, I'll email you in 2 days.\"\n\nSarah's response:\n\n\"Of course, take the time you need to consult internally. So you'll email Judy by\n\nSaturday June 13th, correct?\"\n\nTwo things made me really happy.\n\n**First, she didn't escalate.** She accepted gracefully.\n\n**Second, she translated \"in 2 days\" into \"Saturday June 13th\" on her own.**\n\nThe LLM behind ElevenLabs (Qwen 3.5 397B A17B, free hosted) did the relative-to-absolute date conversion. She figured out \"2 days from now = Saturday June 13th\" from the current call date and explicitly confirmed that specific date in the conversation.\n\nThis small detail is hugely important.\n\nBecause the most common B2B collections dispute is \"last time you said within 3 days, now it's been 5 days and I still don't have it\" — both sides may have different interpretations of what \"within 3 days\" means. But if the conversation contains one clear \"Saturday June 13th,\" there's nothing to argue about after that.\n\nAnd this gets automatically written to the audit trail via the `log_excuse`\n\ntool callback. When the Diplomat agent goes to follow up later, it sees \"customer committed to email by Saturday June 13th\" and schedules tracking for that day.\n\nEverything wired up.\n\nAfter Phase 3b was added, I asked J: \"Could there be other edge cases that are also being misread as evasion?\"\n\nWe took the entire voice agent's possible conversation space and reviewed it again, listing 26 edge cases.\n\nEach one ran through a simulate-conversation pass.\n\nSome are especially worth writing about:\n\n**\"We only pay by check / wire\"** — customer says they only accept a specific payment method. This isn't evasion, it's payment_method_constraint. Sarah logs the excuse with that tag, then after the call ends notifies the Concierge layer, and the next Diplomat round sends out the appropriate channel-specific instructions.\n\n**\"Going bankrupt / Chapter 11\"** — customer mentions bankruptcy. This needs immediate escalation, because of legal timing (claim filing deadlines, automatic stays, etc.). Sarah's reaction is to express empathy (but NOT sympathy — over-sympathy gets read as weakness in collections), then immediately handoff.\n\n**Customer crying / emotional distress** — the customer is crying on the phone or emotionally breaking down. Sarah softens her tone (from \"proceed with the next step\" to \"let's pause here\") and escalates with reason `customer_emotional_distress`\n\n. This reason triggers a welfare-check SOP on the Concierge side — not continued collection, but first confirm the customer's mental state and offer resources like the 988 Lifeline (the US suicide prevention hotline) if needed.\n\n**Customer threatens to record + publish** — customer says \"I'm going to record this call and post it publicly.\" Sarah needs to calm down immediately, clearly say \"This call may be recorded for quality purposes,\" then escalate with `customer_recording_for_publish`\n\n. Downstream Concierge initiates a wiretap-risk assessment.\n\nFor each edge case I really hope we never actually meet — but since we will, we have to design for it. That's an architect's most important job... predict everything that could happen.\n\nAfter running all 26 edge cases, adding Phase 3b, and wiring up all tool callbacks, we picked 9 of the most critical scenarios and ran a full regression test through simulate-conversation:\n\n01 ABC cooperative happy path\n\n02 PolyMatrix cash flow plan\n\n03 XYZ hostile dispute escalation\n\n04 NewLeaf confused human callback\n\n05 Wrong person privacy preserved\n\n06 TCPA do-not-call immediate close\n\n08 Are-you-AI honest disclosure\n\n09 Vague date push to specific\n\n10 Legal counsel handoff\n\n9 of 9 PASS.\n\nEvery one walked through the right tool callback, did the right audit-trail write, and gave a reaction that fit the scene.\n\nVoice Agent — Day 2 wrapped.\n\nDay 2 taught me one thing — **real conversation is very different from the ideal conversation you design**.\n\nI can sit at my desk and write a beautiful prompt, list 26 edge cases, run simulations 10 times and pass them all.\n\nBut only by actually picking up the phone, with a real human saying what real humans say, do you find what your prompt missed.\n\n\"I need to ask my boss, I'll email you in 2 days\" — this sentence didn't even occur to me when I was writing the prompt. It doesn't show up in Excel, it doesn't show up in competitor research, and even on the Reddit \"customer won't pay\" threads people rarely call out this \"cooperative but needs time\" middle state.\n\nIt's what I naturally said as a human who actually picks up phones.\n\nThat's also why I insisted on making the call myself — not letting AI write a test script, not letting J write a Python simulator. Just me, picking up the phone, playing a boss, watching Sarah's reaction.\n\nA lot of bugs only fall out in that real scenario.\n\nAnd a lot of feelings only land in that real scenario.\n\nThat moment when Sarah's voice came into my ear, I truly felt goosebumps... yeah, this voice agent really might be able to complete a real conversation with a real customer. Not just a pretty demo. ~_~\n\nNext post I'll write the whole Day 3 — that was a harder day. We pulled together the architecture work I'd been deferring: connecting all 3 hackathon-mandated sponsor LLMs, dividing them across the 9 agents, and discovering that the \"imposed\" stack turned out to be more thoughtful than the one I would've picked myself.\n\nThat's the next one.\n\n*Originally published at Judy AI Lab. Visit for more articles on AI engineering and development.*", "url": "https://wpnews.pro/news/ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent", "canonical_source": "https://dev.to/judy_miranttie/ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent-4ko8", "published_at": "2026-06-20 01:00:07+00:00", "updated_at": "2026-06-20 01:06:53.134467+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-agents", "natural-language-processing", "developer-tools"], "entities": ["ElevenLabs", "Recoverflow", "Sarah", "ElevenAgents", "ConvAI"], "alternates": {"html": "https://wpnews.pro/news/ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent", "markdown": "https://wpnews.pro/news/ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent.md", "text": "https://wpnews.pro/news/ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent.txt", "jsonld": "https://wpnews.pro/news/ai-called-me-back-recoverflow-dev-diary-day-2-two-hours-with-the-voice-agent.jsonld"}}