{"slug": "improving-caller-identification-for-voice-ai-agents", "title": "Improving caller identification for voice AI agents", "summary": "A production voice AI system for residential property-management companies found that 43% of callers were not pre-identified by phone number, and of those, the name-and-address lookup succeeded only 5 out of 70 times, meaning 19 out of 20 callers were not found. The failures stem from speech-to-text errors that standard fuzzy matching techniques like substring, phonetic, and trigram matching cannot handle, prompting the need for improved caller identification methods.", "body_md": "A note on data and privacy.Results here come from a production voice AI system serving residential property-management companies. The participating management companies are anonymized. Every caller name and street address used as an example — Kari, Whispering Pines Ct., Anika Ravi Shankar — is synthetic, chosen to illustrate a type of transcription error rather than to identify any real person.\n\nOur voice agent answers thousands of calls a day for residential property-management companies. Homeowners call about assessments, violations, architectural requests, account access. The agent picks up, listens, and helps.\n\nAt least, that’s the design. Before it can help anyone, it has to answer a more basic question: whose account is this?\n\nWhen someone calls from a phone number we have on file, that question answers itself in milliseconds. But a lot of people don’t. They’re calling from a work line, a spouse’s phone, a number they changed two years ago and never updated with their association. For those callers, the agent has to ask for a name and address and match what it hears against the account database.\n\nWe pulled 161 production calls and classified how identification went. Forty-three percent of callers weren’t pre-identified by phone number and had to go through the name-and-address lookup.\n\nOf those, the lookup succeeded five times out of seventy.\n\nNineteen out of twenty callers who needed the agent to find them by name and address were not found. That single failure mode was doing more damage to the caller experience than everything else we were working on.\n\n## Why this is harder than it sounds\n\nThe problem isn’t the database lookup. The problem is that by the time the text reaches the database, it isn’t what the caller said.\n\nSpeech-to-text is very good and still wrong often enough to matter. Accents, background noise, a weak cell connection, a child in the room, someone talking while driving. A caller named **Kari** arrives as **Carry**. A street called **Whispering Pines Ct.** comes through as **Whisperingpinescourt**. Neither of those matches anything in the database, so the agent asks the caller to repeat. Then it asks them to spell it. By the time the account surfaces — if it ever does — you’ve spent two minutes of someone’s patience before answering a single question.\n\nThe obvious response is fuzzy matching. We tried the standard toolkit, and it’s worth being specific about why each piece fails, because the failures aren’t random.\n\n**Substring matching** — the SQL `LIKE`\n\noperator — is what we started with. It looks for records containing a specific run of characters. That works when part of your input is exactly right and in the right order, which was a fine assumption when people typed their names into forms. It’s the wrong assumption when the characters themselves are wrong. `LIKE '%Kari%'`\n\nfinds nothing when the transcript says Carry.\n\n**Phonetic matching** like Soundex matches words that sound alike. It fails on the cases people actually produce. Someone who says “Bob” doesn’t match a record stored as “Robert.” A badly split address like “Breakingdotcourt” for “Breaking Dawn Ct.” either misses entirely or returns so many candidates the agent can’t choose between them.\n\n**Trigram matching** — PostgreSQL’s `pg_trgm`\n\n— is the sophisticated option, and this is where it gets interesting. It chops each string into overlapping three-character sequences and scores the overlap. It handles spacing and word-boundary errors far better than `LIKE`\n\n. But it has a specific, structural weakness: each word is padded so its first letter participates in three separate trigrams. Corrupt the first letter and you destroy most of the set.\n\nKari against Carry shares **zero** trigrams. Similarity: 0.00. Evon against Yvonne shares exactly one, giving 0.09. The conventional match threshold is 0.3.\n\nThat’s not a tuning problem. Transcription errors in names hit the leading consonant constantly — it’s the least redundant part of a spoken word — so trigram similarity fails hardest on exactly the cases we most needed it to handle.\n\nEvery one of these tools assumes a kind of regularity that spoken, transcribed input doesn’t have. We needed something that could reason about *how* speech gets mangled.\n\n## Using the model as a query generator, not a matcher\n\nHere’s the design decision that mattered most, and it runs against how most people use large language models for this problem.\n\nThe intuitive approach is to let the model do the matching: retrieve candidate records, hand it pairs, ask which ones refer to the same person. That cannot work inside a live phone call. An LLM call per candidate pair, at conversational latency, with someone waiting on the line — the budget is a few hundred milliseconds before people start talking over the agent. There isn’t room.\n\nSo we inverted it. Our model never sees a candidate record. It sees only the transcribed utterance, and its job is to hypothesize what the caller might actually have said.\n\nGiven the spoken name “Anika Ravi Shankar,” it produces the transcription as-is, a merged form (AnikaRavi Shankar), each component alone, and a compound-surname guess (Ravishankar). Then it compiles all of those hypotheses into a *single* compound database query — one round trip, not many — combining conditions with OR/AND logic. It picks between two strategies depending on what it’s looking at: a broad OR across variations for names and misspellings, or an AND across component words for addresses and partial information.\n\nAdjudication is left to deterministic edit-distance ranking. Levenshtein distance normalized to a 0–100 similarity score, with the agent acting on the number: above 80 it treats the match as found and verifies one more detail, 60 to 79 it asks the caller to confirm, below 60 it asks for something different.\n\nThe result is one LLM call, issued in parallel with database access, doing the part of the job that requires reasoning about speech. Everything downstream is cheap and deterministic.\n\nWe also gave the agent memory across turns. When the first search doesn’t produce a confident match and it has to ask for more information, it passes along what was already tried — so the model adjusts strategy instead of repeating a failed approach.\n\n## What happened\n\nWe built a labeled benchmark from real accounts: 83 accounts across four management companies, 1,156 transcription-error variations spread across twelve distinct error types, with the correct answer known for every case.\n\nTop-result accuracy went from **24.0% to 39.1%**.\n\nThat headline hides the more useful finding, which is that names and addresses are two different problems wearing the same coat.\n\n**Addresses** have structure — a number, a street, a suffix — and the approach exploits that structure well. Top-result accuracy went from 28.6% to 54.7%, a gain of 26 points that held up at every company we tested. The biggest wins came exactly where character matching had been helpless: numbers spoken as words (“one four two Maple Lane”) improved by 50 points, abbreviations by 28, spacing errors by 18.\n\n**Names** are messier, and the honest result is thinner. Top-result accuracy improved only 4 points, from 19.4% to 23.4%. One error class regressed outright: homophones, where what the caller says sounds identical to the right answer but isn’t. “Georgetown Circus” for “Georgetown Circle.” This is the boundary of the approach, and it’s a clean one. Generating more spelling variations cannot help here, because the spoken and stored forms don’t differ in spelling at all — they differ in meaning while sounding the same. Query expansion solves distortions of form. It does not solve substitutions of meaning.\n\n**Latency** was the obvious worry, and it didn’t materialize. Because the model call runs in parallel with database access, lookups came back at equal or better speed than before — at one company, 542 milliseconds faster.\n\n## Takeaways\n\nThe specifics here are about phone calls and property records, but the shape of the problem is common. Any time input reaches you through a lossy channel — transcribed speech, scanned documents, a search box someone typed into on a phone — you are matching a corrupted string against clean records, and the usual tools quietly assume a kind of correctness that isn’t there.\n\nFour things to keep in mind from this experience:\n\n**Measure the failure before you design the fix.** We spent an afternoon classifying 161 calls and it changed what we built. Until we had the 5-out-of-70 number, identification was one item on a list of things that could be better. Afterward it was obviously the only thing worth working on.\n\n**Treat noisy input as evidence, not as a query.** The instinct is to clean up the input and then search for it. The better move is to accept that you don’t know which part is wrong, generate the plausible alternatives, and search for all of them at once.\n\n**Let the model do the part that needs judgment, and nothing else.** Ours never sees a database record or decides which match wins. It generates hypotheses; deterministic code does the ranking. That split is what kept the whole thing fast enough to run inside a live conversation, and it also means the behavior you can’t easily predict is confined to one step you can inspect.\n\n**Break the metric down before you believe it.** Our headline number was a 15-point gain. Underneath it were a 26-point gain on addresses, a 4-point gain on names, and one category that got worse. The aggregate would have told us we succeeded. The breakdown told us what to do next.\n\n*This work has been submitted to IAAI-27.*", "url": "https://wpnews.pro/news/improving-caller-identification-for-voice-ai-agents", "canonical_source": "https://engineering.myhoai.com/posts/improving-caller-identification-for-voice-ai-agents/", "published_at": "2026-08-24 00:00:00+00:00", "updated_at": "2026-08-24 19:12:58.799953+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/improving-caller-identification-for-voice-ai-agents", "markdown": "https://wpnews.pro/news/improving-caller-identification-for-voice-ai-agents.md", "text": "https://wpnews.pro/news/improving-caller-identification-for-voice-ai-agents.txt", "jsonld": "https://wpnews.pro/news/improving-caller-identification-for-voice-ai-agents.jsonld"}}