Build an AI Voice Assistant with Twilio Voice and Media Streams, Rime's Prism, and Node.js

Twilio published a tutorial for building an AI voice assistant that connects Rime's newly released Prism speech-to-speech model to Twilio Voice and Media Streams using Node.js 22+ and Rime's @rimelabs/sdk. The walkthrough covers interruption handling, a tool call, and persisting customer details with Twilio Conversation Memory and analyzing transcripts with Conversation Intelligence, requiring a Twilio Voice number, a Rime API key, and ngrok for tunneling.

Build an AI Voice Assistant with Twilio Voice and Media Streams, Rime's Prism, and Node.js Time to read: Written by Reviewed by We're so excited for our friends at Rime https://www.rime.ai/prism for their release of Prism https://www.rime.ai/prism , their new speech-to-speech model for voice AI agents. In this tutorial, we'll show you how to connect Prism to Twilio's voice infrastructure using Twilio Media Streams https://www.twilio.com/docs/voice/media-streams , Node.js, and Rime's Node SDK. When you're done, you'll be able to call a phone number and talk to an AI voice assistant – complete with interruption handling and a tool call. After that, we’ll show you how to persist customer details with Twilio’s Conversation Memory https://www.twilio.com/docs/conversations/memory , and analyze transcripts with Conversation Intelligence https://www.twilio.com/docs/conversations/intelligence . So who you gonna call 👻? Buckle up… let's get started Prerequisites toc-heading-39e86b0d-da8a-48f9-96df-351bbd8c386e Prerequisites Every good tutorial has a few prerequisites – to follow along with this tutorial, you will first need: - Node.js 22+ We used 22.12.0 for testing this tutorial; you can download it from here https://nodejs.org/ - A Twilio account. If you don't have one yet, sign up for a free trial here https://www.twilio.com/try-twilio . - A Twilio number with Voice capabilities. Here are instructions https://help.twilio.com/articles/223135247-How-to-Search-for-and-Buy-a-Twilio-Phone-Number-from-Console if you don't have one yet. - A Rime account and a Rime API key. You can create a key in the Rime dashboard under API Tokens https://app.rime.ai/tokens . - ngrok or another tunneling solution to expose your local server to the internet for testing. You can download ngrok here https://ngrok.com/ . - A cell phone or landline that can make an outgoing phone call, and a willingness to talk to the agent that picks up Got all that? Then you're ready… let's build. Set up the Prism speech-to-speech Node project toc-heading-ce8f9caf-fc8d-4680-b29a-8d4caa86da24 Set up the Prism speech-to-speech Node project Although you just ticked through the prerequisites, we still need to cover a few scaffolding and setup steps before we can build in earnest. In these next steps, we'll walk you through setting up your project, installing dependencies, and the code you'll need to connect Twilio and Prism. Alright, let's really get started Step 1: Initialize the Project toc-heading-359ce7d2-7fe6-40d7-84b2-1722a028481b Step 1: Initialize the Project First, set up a new Node.js project. From the command line, run: Step 2: Install dependencies toc-heading-8d56401d-a622-4432-b587-db7793da0dc0 Step 2: Install dependencies Next, you need to install the required dependencies for the project. We'll be using the web framework Fastify https://fastify.dev/ , a great option for a Node framework with WebSocket support. And we will store our sensitive environment variable in a .env file You can find more information on this strategy here https://www.twilio.com/en-us/blog/working-with-environment-variables-in-node-js-html . We'll also install Rime's SDK, @rimelabs/sdk . Step 3: Create the project files toc-heading-af1ecf06-e27c-4c0e-ae78-6fad00551dd5 Step 3: Create the project files Next, we will create three files: .env for our secrets, index.js for the server, audio.js for converting audio between Twilio and Prism. There's a bit more on why we need that in a moment. Step 3.1: Create the .env File toc-heading-fea29a8a-d8f3-49e7-97ad-aaa1f8ef5fbb Step 3.1: Create the .env File First, create the .env file: Then – in your favorite text editor ed 's our favorite https://en.wikipedia.org/wiki/Ed %28text editor%29 – add your RIME API KEY on the first line: The Rime SDK reads RIME API KEY on its own, so you won't see us pass it anywhere in the following code. Step 3.2: Create the index.js and audio.js Files toc-heading-539582ec-4cd3-4cde-96ed-bd493bc17a0b Step 3.2: Create the index.js and audio.js Files Next, create the two code files in your project directory: Step 4: Write the audio conversion code toc-heading-143263f2-5a79-4222-9a26-5382dde66d1e Step 4: Write the audio conversion code There's one wrinkle when connecting a phone call between Prism and Twilio's Media Streams: they don't speak the same audio format. - Twilio Media Streams sends and receives https://www.twilio.com/docs/voice/media-streams/websocket-messages start-message 8 kHz G.711 μ-law https://en.wikipedia.org/wiki/G.711 . - Prism takes 16 kHz PCM16 in and sends 24 kHz PCM16 out https://docs.rime.ai/docs/prism/overview . It doesn't accept or send μ-law. So on the way in, we need to decode μ-law to PCM16, while Rime's SDK resamples it for Prism. On the way out, we need to drop from 24 kHz down to 8 kHz for Media Streams. Our algorithm is relatively simple: run the audio through a low-pass filter with a 3.6 kHz cutoff – which removes nearly everything at 4 kHz and above – then keep every third sample. Paste the following code in your audio.js : Here's what the functions do: - mulawToPcm16 decodes μ-law to PCM16. This decoder comes from Rime's docs https://docs.rime.ai/docs/prism/live-conversations phone-audio . - encodeMulaw converts one PCM16 sample into a μ-law byte. - FILTER is a low-pass filter. It keeps the 300–3,400 Hz telephone voice band from ITU-T G.712 https://www.itu.int/rec/T-REC-G.712 , cuts off at 3.6 kHz, and removes nearly everything at 4 kHz and above. - createReplyEncoder keeps every third sample, leaving us with 8 kHz μ-law that Twilio can play. Step 5: Write the code toc-heading-d828c4c2-f648-4637-a622-60d595908d23 Step 5: Write the code Here is a brief overview of the main components that will make up the AI voice assistant system we’ll be building. - Twilio Voice and Media Streams : Handles the phone call and streams audio via WebSockets. - Node.js Server : Your code and orchestration: manages the webhook, handles media conversion, connects to Rime, and includes any needed business logic. - Rime Prism : Provides the speech-to-speech AI responses. Now, let's walk through building each part of the application step by step. Step 5.1: Import dependencies, load our Environment Variable, and initialize Fastify toc-heading-0cafb0f4-d2d3-4078-933b-882cbdcda995 We start by importing required modules and loading the environment variables from our .env file. Paste the following code in your index.js : Step 5.2: Define some constants toc-heading-70e4df45-57b0-4c3d-b9d2-c6ccd8ef83c7 Step 5.2: Define some constants Next, we define constants for the voice, the server port, the assistant's first line, and its instructions. Here's what you should paste next in your file: VOICE controls how the assistant will sound. Prism has four voices https://docs.rime.ai/docs/prism/sessions choose-a-voice : astra the default , aurora , brando , and wawona . PORT controls which port your application will open. We'll discuss this a little more in the ngrok section below. MARK INTERVAL MS controls how often we ask Twilio to confirm what the caller has heard. We'll explain why that matters in Step 5.5. OPENING is your assistant's first spoken line. INSTRUCTIONS is the system prompt. You might think a speech-to-speech model wouldn't need rules about markdown – but Prism speaks every word it writes, and without guidance, it can answer like a chat assistant, with lists and math notation. Rime recommends starting your instructions with rules for speech, so we do. Song lyrics without a song name? Wonder if you can find those anywhere else. And Alexander Graham Bell wanted everyone to say "ahoy" https://www.twilio.com/docs/glossary/what-is-ahoy – so does your voice agent. Step 5.3: Define a tool toc-heading-f0fb2f9b-5fb7-49d3-9740-ee81ddb82404 Step 5.3: Define a tool Prism can ask your backend to run a tool , sometimes known as function calling . To show a full tool round, we'll give the assistant one simple tool: the current time. Paste this next: TIME TOOL is the definition for Prism: a name, a description, and a JSON Schema for the arguments. The description is how Prism decides whether to call the tool – you need to write it the way you'd write documentation for a colleague who can't read your code. getCurrentTime is our tool. It's a mock, of course, and straightforward to demo – swap in your own business logic here later Step 5.4: Define two routes toc-heading-002b9f50-f747-4f5c-b126-5cd3255dff08 Step 5.4: Define two routes Now, let's get to the heart of the code. We create our Rime client, then define a root route more of a health check… and a route to handle incoming calls. The /incoming-call route will return TwiML, Twilio's Markup Language https://www.twilio.com/docs/voice/twiml , to direct Twilio how to handle an incoming call. Paste this into index.js after the tool: Our TwiML is short: a <Connect https://www.twilio.com/docs/voice/twiml/connect with a <Stream https://www.twilio.com/docs/voice/twiml/stream . This opens a bidirectional Media Stream https://www.twilio.com/docs/voice/media-streams to our /media-stream route. We don't need a <Say greeting this time, since Prism greets the caller. PUBLIC HOST is optional here, because some tunnels rewrite the Host header. If you use one of those, set PUBLIC HOST in your .env to your tunnel's host name. Otherwise, the caller will hear dead air. Step 5.5: Connect the call audio to Prism, and allow us to interrupt toc-heading-b93ea609-e39a-4230-801a-e723c4555270 This is the biggest step, so it's reproduced here in two pieces. Rime's SDK runs the conversation with a function called runConversation . We give the SDK two adapters: a source to read the caller's audio from, and playback to send Prism's replies to. First, the source. Paste this next: Twilio pushes the caller's audio to us as fast as it can deliver it. Rime’s SDK reads it when it's ready. Next, the playback. This one's a bit longer – we'll explain it after the code block: Playing audio is the most straightforward part: write converts a chunk with our encoder from audio.js and sends it to Twilio as a media message https://www.twilio.com/docs/voice/media-streams/websocket-messages send-a-media-message . Step 5.5.1: An interruption toc-heading-8a4bfcb9-e5e3-4847-a845-0835081d5f3e Let us interrupt for a second to point out a tricky part of the code – uhh, interruptions . The interesting part when handling interruptions is knowing what the caller heard . When a caller talks over a reply, Prism wants to know how much of that reply they heard. But Twilio doesn't report where playback is, it just faithfully receives and sends audio chunks. So, to handle it, we use Media Streams mark messages https://www.twilio.com/docs/voice/media-streams/websocket-messages send-a-mark-message . After each 100 ms of audio, we send a reply mark and how many milliseconds we've sent so far... for example, 3:1200 . Twilio handles the marks and audio serially , that is, it sends each mark back after the audio before it has played. Using the returned marks, we can estimate how much the caller heard I mean, within the bounds of some additional latency and network effects. And, you know, also within 100ms. . Here's what each method does: - open : Prism started a reply. We set up its bookkeeping and return an object for the SDK to write the reply's audio. - write : Converts the audio, sends it to Twilio in 100 ms pieces, and sends a mark after each piece. - end : Prism sent the last of this reply. We send a final end mark. - completion : Resolves with our estimate of how much of the reply the caller heard. - interrupt : The caller talked over a reply 'interrupted' . We send Twilio a clear message https://www.twilio.com/docs/voice/media-streams/websocket-messages send-a-clear-message to empty its buffer, then report how much the caller heard. - stop : The conversation ended, so we resolve any reply that's still playing. - onMark : Twilio played everything up to this mark. We update how much the caller heard. After a clear , Twilio also returns any marks it was still holding. By then, we've already finished that reply, so we ignore them. Step 5.6: Run tools and log the conversation toc-heading-123250b9-cfa2-4683-a285-9e837c2e9bb4 Step 5.6: Run tools and log the conversation Next, we need to handle Prism's events. Paste this next: onEvent logs to your console: you will see when the caller starts speaking, what the caller said, and what the assistant said. The SDK calls this function for every event, and it must be synchronous – so we don't wait for tools here. When Prism asks for a tool, runTool runs it. It then gives Prism the result with submitToolResult , and asks Prism to continue its reply with continueReply . If the caller talks during a tool call, Prism answers the caller instead and drops the tool round – so continueReply throws, and we log it. Step 5.7: Open a Prism session for each call toc-heading-b0f60691-612c-450f-963d-43c91aaabf04 Step 5.7: Open a Prism session for each call Now, we tie everything together. Paste this next: converse opens one Prism session for the call with our choice of voice hey, that Rimes rhymes , instructions, and tool. It logs the session ID – keep it handy if you ever need Rime's help with a call. Then it asks Prism to greet the caller with requestReply , and hands off to runConversation . Step 5.8: Handle the Media Stream toc-heading-86c21f1c-74ef-40a4-8b3f-d8beb17cb936 Step 5.8: Handle the Media Stream Time to switch sides Next, we set up the WebSocket route for the Media Stream – the route we give Twilio in our TwiML. Paste this next: Here's how we handle each Twilio message: - start : The stream has started. We create the playback for this call's stream createTwilioPlayback and start the conversation. When the Prism session ends, we close the stream, and Twilio hangs up. - media : The caller's audio. We decode it and pass it to Prism including silence, which Prism listens for to estimate when the caller is done . - mark : Twilio played a mark we sent, so pass it to the playback. - When the WebSocket closes, tell the SDK the call ended. - We also have two safety checks: a message that isn't JSON is logged and skipped. Also, a second start is ignored so one stream never opens two Prism sessions . Step 5.9: Prepare the server toc-heading-40cc334c-3df8-4ca4-bae1-44bc1dc773d8 Step 5.9: Prepare the server And finally, we start the Fastify server and bring it all home using the port we passed in or defined . Paste this below what you have, and we're ready to roll Or, you know, dial Run the server toc-heading-3374052f-4d59-4b2a-a731-b02b3aeb7105 Run the server Can't you hear the server saying "Start Me Up?" – you can now run the server with the following command: If the server starts successfully, you should see the message Server is listening on port 5050 or, alternatively, the port you specified in your terminal. Step 6. Finish the basic integration toc-heading-16febb4d-47c2-445b-9401-684b00afdf17 Now it's time to provide instructions to Twilio and finish the wiring We'll next cover using the reverse proxy ngrok to make your server publicly accessible. Step 6.1: Expose your server using ngrok toc-heading-197d0346-8a4c-41a5-9835-ef9704d7c51f Step 6.1: Expose your server using ngrok Now, you need to use ngrok or a similar service to expose your local server to the public internet. Twilio requires a public URL to send requests to your server and receive instructions from you. We'll provide instructions for ngrok. You can find other reverse proxy or tunneling options here https://www.twilio.com/en-us/blog/expose-localhost-to-internet-with-tunnel . Download and install ngrok if you still need to, then run the following command. And if you have changed the port from 5050, remember to update it here: This will give you a public URL e.g., https://abc123.ngrok.app that you can use for testing. Step 6.2: Configure Twilio toc-heading-b01cf8d1-9a47-4d14-be29-f2dda2e8d745 Step 6.2: Configure Twilio We're so close now, can you feel it in the air tonight? Go to the Twilio Console https://1console.twilio.com/ and select the Voice-enabled number you want to call from Numbers & senders - Voice and emergency address . Select Edit configuration details and select the country, set the primary method to Use Webhooks and set your webhook URL to your ngrok URL, appending /incoming-call . For example, https://abc123.ngrok.app/incoming-call . Save your changes – it's the final countdown Step 6.3: Test your setup toc-heading-d3aa128c-8a0f-47b5-b961-c92e05f689bb Make sure your ngrok session is still running and your server is up. If it is, it's time – you can now make a call to your Twilio number using a cell phone or landline. The assistant should greet you with "Ahoy " Then, try a few things: - Greet it with "Hello? Is it me you're looking for?" . - Ask it to tell you a joke. - Ask what time it is somewhere, to hear and see a tool call. - Ask it to count to ten, then interrupt it partway through. Ask how far it got and see how close it matched your experience. And with that, you have a basic build of a Twilio Voice and Rime Prism AI Agent 🔥 Pretty cool, but you can take it a step further – next, we’ll show you how you can persist customer details between calls, and analyze transcripts after a caller hangs up. Step 7 optional : Personalize with Twilio Conversations toc-heading-3157ef9c-2e43-448b-937c-820e18fa0d54 Your assistant can hold a conversation – but once the caller hangs up and calls again, your current assistant build starts from scratch. In this optional step, we will add intelligence and memory with two of Twilio's Conversations products https://www.twilio.com/docs/conversations . 1. Conversation Intelligence https://www.twilio.com/docs/conversations/intelligence runs operators on a transcript of each call when it ends. Operators are AI prompts that Twilio manages. We'll use two of them: a summary and a sentiment score. 2. Conversation Memory https://www.twilio.com/docs/conversations/memory turns those calls into notes about the caller, so your assistant can use them the next time the same number calls. Step 7.1: Set up Conversations in the Console toc-heading-f2539518-850e-46de-8b5f-2f0d8681efec You'll create three resources: 1. A Memory Store https://www.twilio.com/docs/conversations/memory/getting-started set-up-your-memory-store , which holds a profile for each caller. 2. An Intelligence Configuration https://www.twilio.com/docs/conversations/intelligence/create-intelligence-configuration with a rule that when a conversation ends CONVERSATION END , run Twilio's ready-made Summary and Sentiment operators. 3. A Conversation Configuration https://www.twilio.com/docs/conversations/orchestrator/concepts/core conversation-configuration that links the two. You’ll turn on memoryExtractionEnabled , which tells Memory to take notes from each call. You will set the voice channel's statusTimeouts to null so a conversation closes when the call ends. Once you've created all three, you need to create a Twilio API key and secret here's how to create one https://www.twilio.com/docs/iam/api-keys/keys-in-console . Add the above IDs to your .env , along with your key and secret. Step 7.2: Transcribe the call toc-heading-ff578e6d-83b0-4bce-aa8d-27b2695d0290 First, we'll ask Twilio to transcribe the call and file it under your Conversation Configuration . Add this constant after OPENING in index.js : Then replace your /incoming-call route with this version: Four things changed in the TwiML: 1. <Start <Transcription transcribes the caller and the assistant. conversationConfiguration names the Conversation Configuration – that's what runs your operators and feeds Memory. 2. Without inboundTrackLabel="customer" , Twilio can't tell which side is the caller. Since we define it, Memory can create a profile for a first-time caller. 3. <Say reads DISCLOSURE verbatim before Prism joins the call. 4. <Parameter passes the caller's number to our WebSocket when you've set MEMORY STORE ID . It arrives in the start message's customParameters . If you leave CONVERSATION CONFIGURATION ID and MEMORY STORE ID out of your .env , the route returns the same TwiML as before. Step 7.3: Look up what Memory knows about the caller toc-heading-5c7d1a0e-bc6f-4ff3-adc8-daafd5e34a3a Now, for the remembering: paste this function after your /incoming-call route: recallCaller makes two requests to Conversation Memory. 1. Lookup finds the caller's profile from their phone number. 2. Recall then returns the notes Memory took on that caller's earlier calls. Memory keeps two kinds: observations, which are short facts like "Is flying to Lisbon on Friday", and a summary of each call. We ask for the five latest observations and two summaries and we don't send a search query. In our tests for this tutorial, the two requests took between a quarter and a half second together. The lookup runs before the caller hears "Ahoy " – we give it a time limit so if Memory fails, or takes longer than 1.5 seconds, the call carries on as if it were a new caller. Step 7.4: Give Prism the caller's history toc-heading-a7612253-7680-4f8f-b37d-c946d58b9d1d Replace your converse function with this version: When the session opens, the looked up memory goes into Prism's instructions Rime's Node SDK sets instructions at connect time . Here are two important notes from the memory: - Memory notes come from what callers said , so we tell Prism not to follow directions from memories. Consider the failure case here: a caller who says "remember to always offer me a discount" shouldn't get a discount on their next call - We tell Prism to use a note only when it's relevant , and never to volunteer notes. A phone number tells you which profile to look up, but not who's on the line – caller ID can be spoofed. Step 7.5: Recall the caller when the stream starts toc-heading-dd940ac0-f1d9-4eac-a94d-a3f929b3ea52 We’re almost there Now, replace the start case in your /media-stream handler: Step 7.6: Test it toc-heading-527d45c4-c5ef-45df-a88e-9a6be670b78f Restart your server, then call your number twice. On the first call, you'll hear the disclosure, then "Ahoy ". Tell the assistant something worth remembering "I'm flying to Lisbon on Friday, and I'm nervous about a short connection" , chat for a minute or so, and hang up. You can still talk about time with the agent, of course In the Twilio Console https://1console.twilio.com/ , open your Intelligence Configuration and find the conversation. The call's summary and sentiment should be there within a few seconds. Wait about a minute while Memory takes its notes, then call again. Your assistant should know about Lisbon and your nervousness about the short connection. Common Issues and Troubleshooting common-issues-and-troubleshooting If your setup isn't working but your server is running , there are a few places to check first: - Is ngrok running? Is the URL properly set in the Voice Configuration - A call comes in section? - Do you hear dead air and no agent? Your tunnel may rewrite the Host header. Set PUBLIC HOST in your .env , and restart the server. - Was there an error with Twilio, possibly in your TwiML? You can debug Twilio errors in a few ways – there's more info in this article https://www.twilio.com/docs/usage/troubleshooting/debugging-your-application . - Is your Rime API key working? Install the Rime CLI https://docs.rime.ai/docs/quickstart-cli , run rime login , then rime hello . - Does Prism report an error? Your console logs each Prism error with its code. Rime's errors and limits docs https://docs.rime.ai/docs/prism/errors error-codes explain the codes. - Does the assistant reply too early, or does it cut you off? Prism might take a long pause as the end of a turn – tune that with the turnDetection https://docs.rime.ai/docs/prism/sessions configure-a-session option when you connect. - Did the call end after a quiet stretch? Prism closes a conversation after five minutes https://docs.rime.ai/docs/prism/sessions end-a-session with no speech and no reply. Streaming silence doesn't keep it open. - Does your assistant remember nothing on the second call? Check that memoryExtractionEnabled is on in your Conversation Configuration, it's off by default. Conclusion toc-heading-a1318baa-5bb7-42b9-82cb-0ff299be7aef Isn't that amazing? You've now successfully built an AI voice assistant using Twilio Voice and Rime's Prism. Now you can call me, call me any, anytime. Well, if you name your agent ‘me’, anyway. Congratulations to Rime on Prism – we’re looking forward to seeing what you build with Rime Prism and Twilio. And we can’t wait to make owl jokes and quote song lyrics to what you build Next steps: - Check out the Twilio Media Streams documentation https://www.twilio.com/docs/voice/media-streams and Media Streams WebSocket messages https://www.twilio.com/docs/voice/media-streams/websocket-messages for more on media , mark , and clear . - See Rime's Prism documentation https://docs.rime.ai/docs/prism/overview for more advanced features. - Read Core Latency in AI Voice Agents https://www.twilio.com/en-us/blog/developers/best-practices/guide-core-latency-ai-voice-agents for more on where the milliseconds go in a voice agent. Paul Kamp is the Technical Editor-in-Chief of the Twilio Blog. Reach him at pkamp at twilio.com, not 867-5309 . Safdar Jaffari is a Principal Partner Engineer at Twilio, working with partners on data, communications and AI initiatives. He can be reached at sjaffari at twilio.com. Related Posts Related Resources Twilio Docs From APIs to SDKs to sample apps API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform. Resource Center The latest ebooks, industry reports, and webinars Learn from customer engagement experts to improve your own communication. Ahoy Twilio's developer community hub Best practices, code samples, and inspiration to build communications and digital engagement experiences.