How to build multilingual AI agents for customer support

Twilio published a guide for building multilingual AI voice agents on its ConversationRelay product, which handles speech-to-text and text-to-speech while letting developers connect the LLM of their choice. The guide instructs developers to add one <Language> element per language a call may use, each mapping a language code to its own text-to-speech provider, voice, transcription provider and speech model, and to set the speech-to-text language to "multi" with Deepgram as the transcription provider so the system identifies the language of each utterance automatically. Twilio notes that all three layers of a phone conversation — speech-to-text, the LLM and text-to-speech — must switch together, since an LLM that handles Spanish still fails if speech-to-text is listening for English.

How to build multilingual AI agents for customer support Time to read: Written by How to build multilingual AI agents for customer support introduction Language isn’t always linear. A customer might initiate a call in English but switch to Spanish midway through the conversation. It happens, but most voice agents keep right on answering in English. The caller repeats themselves or perhaps returns to a non-native English, or they ask for a human—either way, your AI agent has let them down. It doesn’t need to be this difficult, though. Nor do you need to hire dozens of bilingual agents or translate your entire help center. AI agents make multilingual customer support scalable for businesses of all sizes and budgets. A single agent can serve customers in many languages, but it needs to be able to detect when a caller switches languages and follow along when it does. Here's how to build an AI agent that detects a customer's language, switches mid-conversation, and remembers the preference for next time on the phone and in messaging . What is a multilingual AI agent? toc-heading-df6f017d-72f8-45f3-8bef-6d30f33a03dc A multilingual AI agent is an AI agent that detects and responds to customers in more than one language. Smart multilingual agents recognize which language a customer is using and switch its own language to match. This is simpler in messaging and chat functions. Modern LLMs read and write dozens of languages, so a text-based agent can usually respond in whatever language the customer writes in. Easy. But voice is harder. A phone conversation runs through three systems: 1. Speech-to-text https://www.twilio.com/en-us/speech-recognition turns the caller's voice into words 2. The LLM reads https://www.twilio.com/docs/glossary/what-is-a-large-language-model and decides what to say 3. Text-to-speech https://www.twilio.com/docs/voice/twiml/say/text-speech turns the reply back into audio The LLM might handle Spanish perfectly, but if speech-to-text is still listening for English, the transcript it hands the LLM won’t work. All three layers have to switch together. How to build a multilingual AI agent how-to-build-a-multilingual-ai-agent We’ll be using Twilio ConversationRelay https://www.twilio.com/en-us/products/conversational-ai/conversationrelay , which handles speech-to-text and text-to-speech for voice AI agents. It also lets you connect the LLM of your choice. First, you’ll configure languages before you detect them, and then you’ll need to detect them before you switch. 1. Choose speech providers that cover your languages toc-heading-df21c72b-2ae0-433f-9ac4-62c7dcd9b42d Every language your agent might hear needs a speech-to-text provider that can transcribe it and a voice that can speak it. Language availability, coverage, and quality vary depending on your provider, so start with the languages your customers speak, then double-check each one is supported. In ConversationRelay, you add one <Language element for each language a call might use https://www.twilio.com/docs/voice/twiml/connect/conversationrelay . Each element maps a language code to its own text-to-speech provider, voice, transcription provider, and speech model. That means your Spanish voice can come from a different provider than your English one if that's what sounds best for you. Adding a <Language element doesn't activate it, though. It tells ConversationRelay what to use when the conversation switches to that language. 2. Detect the caller's language automatically toc-heading-328cd5da-3984-466e-9389-c81d54b5ae0e ConversationRelay detects a caller's language automatically when you set the speech-to-text language to multi this requires Deepgram as the transcription provider https://www.twilio.com/en-us/changelog/conversationrelay-now-supports-a-configuration-for-automatic-lan . The speech-to-text provider identifies the language of each utterance, and every transcribed message your agent receives arrives tagged with that language in a lang field. Again, that’s per utterance . Your agent doesn't have to guess the language from the words, and it catches the moment a caller switches partway through a call, because the tag changes on the next utterance. - Tags arrive as primary language codes: Detected languages come back as en or es, without a regional suffix like en-US. If your code maps languages to specific voices, account for that. - Text-to-speech can detect too: Setting the text-to-speech language to multi lets ElevenLabs detect the language from the text your agent sends, so the voice follows the reply. If you know which languages to expect, ConversationRelay also supports language hints. With Deepgram's Flux speech model, your <Language elements act as hints that bias recognition toward the languages you've configured. Flux also pairs with a newer attribute, ignoreBackchannel, which filters filler words like “uh-huh” and “okay” so they don’t interrupt the agent mid-reply. Twilio says it covers English, Spanish, French, German, and more. 3. Switch languages mid-conversation toc-heading-8349712a-2b5b-4b14-93ef-d115b0b5a976 ConversationRelay switches languages mid-conversation through a language message your agent sends over the WebSocket connection https://www.twilio.com/docs/voice/conversationrelay/websocket-messages . This changes the speech-to-text language or the text-to-speech language or both without restarting the call. A typical flow looks like this: 1. The caller switches to Spanish. The next transcribed message arrives tagged es. 2. Your agent notices the change. Your code compares the new tag to the current language. 3. Your agent sends a language message. It sets the new transcription and text-to-speech languages. 4. The conversation continues in Spanish. The voice changes to the one you configured for Spanish, and the agent keeps going. For finer control, you can also set the language on individual text tokens your agent sends. That helps when a reply mixes languages, like an English-speaking agent reading out a Spanish street address. 4. Keep the LLM consistent across languages toc-heading-e7403989-519a-44fc-bc4a-35587f4e26f3 Switching the voice is only half the job. The LLM also needs clear instructions about what to do when the language changes. These prompt-design rules can help: - Respond in the customer's current language: Pass the detected language into the agent's context on every turn, and instruct it to answer in that language. - Keep facts language-independent: Order numbers, prices, and policy details should come from your tools and knowledge base unchanged, whatever language the conversation is in. - Decide how to handle mixed input: Some customers switch languages mid-sentence. Tell the agent whether to follow the most recent language or the dominant one. Watch our October 2026 webinar, "One agent, many languages" https://webinars.twilio.com/twilio/apj-level-up-multi-lingual-voice-ai , to learn how to implement these prompt-design and fallback decisions on ConversationRelay. 5. Remember the customer's language for next time toc-heading-8540499e-8933-4ff8-b374-371e1e98d066 When your customer spoke Spanish last week, it’s safe to assume they’ll prefer the same this week. Store the preference. Twilio Conversation Memory https://www.twilio.com/en-us/products/conversational-ai/conversation-memory keeps traits and observations on each customer's profile. Twilio's Agent Connect and Flex handoff tutorial https://www.twilio.com/en-us/blog/developers/tutorials/product/twilio-agent-connect-flex-human-handoff shows once you’ve updated the language trait on the customer's profile to Spanish, the agent communicates with that customer in Spanish. Simple as that. On the next call, your agent reads the trait and starts in Spanish from the first greeting. Multilingual agents across voice and messaging multilingual-agents-across-voice-and-messaging Customers don’t follow a linear customer journey. They bounce between channels. But their language preferences shouldn’t be limited to a single channel. It’s safe to assume by default at least , that when your customer prefers to talk over the phone in Spanish, they’ll probably want to text in Spanish, too. Twilio Agent Connect https://www.twilio.com/docs/conversations/agent-connect routes Voice, SMS, WhatsApp, RCS, and Chat through a single message handler, so one agent with one set of language rules covers every channel. On messaging channels, the LLM reads the customer's language directly from the text. On voice, ConversationRelay handles detection and switching as described above. The language trait on the customer's profile ties it together. A customer who texts in Portuguese on Monday and calls on Thursday reaches an agent that already knows to speak Portuguese. Escalation: Handing off to a human who speaks the language toc-heading-225a15cf-79d6-4b88-9707-5d8ea0472800 The escalation path should build off what your agents have already learned. That means when a Spanish-speaking customer wants to talk to a person, you route them to a Spanish-speaking representative. Agent Connect's built-in handoff tool accepts optional routing attributes, and those attributes surface as TaskRouter task attributes https://www.twilio.com/docs/conversations/agent-connect/escalate-to-human-agent in Twilio Flex https://www.twilio.com/en-us/flex . Pass the detected language as an attribute, and Flex can route the conversation to an agent who speaks it. The customer stays on the same call or chat throughout, and the conversation history travels with the handoff. Meet customers in their language on their channel toc-heading-77929f2c-6a12-4558-8b81-39c9a278df8f Work smarter so your customers work less harder. A multilingual agent lets you meet customer in their language on their preferred channel. Configure your languages, let detection run on every utterance, switch the instant the customer does, and remember the preference for next time. If you’re worried about Voice AI speed, check out our guide to core latency in AI voice agents https://www.twilio.com/en-us/blog/developers/best-practices/guide-core-latency-ai-voice-agents . Frequently asked questions frequently-asked-questions What are the best platforms for building multilingual AI agents that can detect language and switch mid-conversation? Twilio ConversationRelay detects a caller's language on every utterance when set to multi-language mode with Deepgram, and switches speech recognition and voice languages mid-call through a language message. Agent Connect extends the same agent to SMS, WhatsApp, RCS, and Chat. How does an AI voice agent detect which language a caller is speaking? Twilio ConversationRelay uses the speech-to-text provider to identify the language of each utterance and tags every transcribed message with it. Your agent reads the tag and can switch languages as soon as the caller does. Can an AI agent switch languages in the middle of a phone call? Twilio ConversationRelay lets your agent send a language message that changes the speech-to-text and text-to-speech languages mid-call without restarting the conversation. The voice switches to the one configured for the new language. How many languages can a multilingual AI agent support? Twilio ConversationRelay supports the languages offered by its speech-to-text and text-to-speech providers, configured with one language element per language. Coverage and quality vary by provider, so confirm each language your customers speak. How do you route a multilingual customer to a human agent who speaks their language? Twilio Agent Connect's handoff tool passes routing attributes into Flex as TaskRouter task attributes. Include the detected language, and Flex can route the conversation to an agent who speaks it, with the history attached. Related Posts Related Resources Twilio Docs From APIs to SDKs to sample apps API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform. Resource Center The latest ebooks, industry reports, and webinars Learn from customer engagement experts to improve your own communication. Ahoy Twilio's developer community hub Best practices, code samples, and inspiration to build communications and digital engagement experiences.