{"slug": "build-an-ai-voice-assistant-with-twilio-voice-and-media-streams-rime-s-prism-and", "title": "Build an AI Voice Assistant with Twilio Voice and Media Streams, Rime's Prism, and Node.js", "summary": "Twilio published a tutorial for building an AI voice assistant that connects Rime's newly released Prism speech-to-speech model to Twilio Voice and Media Streams using Node.js 22+ and Rime's @rimelabs/sdk. The walkthrough covers interruption handling, a tool call, and persisting customer details with Twilio Conversation Memory and analyzing transcripts with Conversation Intelligence, requiring a Twilio Voice number, a Rime API key, and ngrok for tunneling.", "body_md": "# Build an AI Voice Assistant with Twilio Voice and Media Streams, Rime's Prism, and Node.js\n\nTime to read:\n\n**Written by**\n\n**Reviewed by**\n\nWe're so excited for our friends at [Rime](https://www.rime.ai/prism) for their [release of Prism](https://www.rime.ai/prism), their new speech-to-speech model for voice AI agents.\n\nIn this tutorial, we'll show you how to connect Prism to Twilio's voice infrastructure using [Twilio Media Streams](https://www.twilio.com/docs/voice/media-streams), Node.js, and Rime's Node SDK. When you're done, you'll be able to call a phone number and talk to an AI voice assistant – complete with interruption handling and a tool call. After that, we’ll show you how to persist customer details with Twilio’s [Conversation Memory](https://www.twilio.com/docs/conversations/memory), and analyze transcripts with [Conversation Intelligence](https://www.twilio.com/docs/conversations/intelligence).\n\n*So who you gonna call 👻?* Buckle up… let's get started!\n\n## [**Prerequisites**](#toc-heading-39e86b0d-da8a-48f9-96df-351bbd8c386e)\n\n**Prerequisites**\n\nEvery good tutorial has a few prerequisites – to follow along with *this* tutorial, you will first need:\n\n- Node.js 22+ (We used 22.12.0 for testing this tutorial; you can download it from [here](https://nodejs.org/) )\n- **A Twilio account.** If you don't have one yet,[sign up for a free trial here](https://www.twilio.com/try-twilio) .\n- A Twilio number with **Voice** capabilities.[Here are instructions](https://help.twilio.com/articles/223135247-How-to-Search-for-and-Buy-a-Twilio-Phone-Number-from-Console) if you don't have one yet.\n- **A Rime account and a Rime API key.** You can create a key in the Rime dashboard under[API Tokens](https://app.rime.ai/tokens) .\n- **ngrok or another tunneling solution** to expose your local server to the internet for testing.[You can download ngrok here](https://ngrok.com/) .\n- **A cell phone or landline** that can make an outgoing phone call, and a willingness to talk to the agent that picks up!\n\nGot all that? Then you're ready… let's build.\n\n## [**Set up the Prism speech-to-speech Node project**](#toc-heading-ce8f9caf-fc8d-4680-b29a-8d4caa86da24)\n\n**Set up the Prism speech-to-speech Node project**\n\nAlthough you just ticked through the prerequisites, we still need to cover a few scaffolding and setup steps before we can build in earnest. In these next steps, we'll walk you through setting up your project, installing dependencies, and the code you'll need to connect Twilio and Prism.\n\nAlright, let's  *really* get started!\n\n###  [**Step 1: Initialize the Project**](#toc-heading-359ce7d2-7fe6-40d7-84b2-1722a028481b)\n\n**Step 1: Initialize the Project**\n\nFirst, set up a new Node.js project. From the command line, run:\n\n###  [**Step 2: Install dependencies**](#toc-heading-8d56401d-a622-4432-b587-db7793da0dc0)\n\n**Step 2: Install dependencies**\n\nNext, you need to install the required dependencies for the project. We'll be [using the web framework Fastify](https://fastify.dev/), a great option for a Node framework with WebSocket support. And we will store our sensitive environment variable in a `.env` file (You can find more information on this strategy [here](https://www.twilio.com/en-us/blog/working-with-environment-variables-in-node-js-html)).\n\nWe'll also install Rime's SDK, `@rimelabs/sdk`.\n\n###  [**Step 3: Create the project files**](#toc-heading-af1ecf06-e27c-4c0e-ae78-6fad00551dd5)\n\n**Step 3: Create the project files**\n\nNext, we will create three files: `.env` for our secrets, `index.js` for the server, `audio.js` for converting audio between Twilio and Prism. (There's a bit more on why we need that in a moment.)\n\n####  [**Step 3.1: Create the .env File**](#toc-heading-fea29a8a-d8f3-49e7-97ad-aaa1f8ef5fbb)\n\n**Step 3.1: Create the .env File**\n\nFirst, create the `.env` file:\n\nThen – in your favorite text editor ( [`ed`'s our favorite](https://en.wikipedia.org/wiki/Ed_%28text_editor%29)) – add your `RIME_API_KEY` on the first line:\n\nThe Rime SDK reads `RIME_API_KEY` on its own, so you won't see us pass it anywhere in the following code.\n\n####  [**Step 3.2: Create the index.js and audio.js Files**](#toc-heading-539582ec-4cd3-4cde-96ed-bd493bc17a0b)\n\n**Step 3.2: Create the index.js and audio.js Files**\n\nNext, create the two code files in your project directory:\n\n###  [**Step 4: Write the audio conversion code**](#toc-heading-143263f2-5a79-4222-9a26-5382dde66d1e)\n\n**Step 4: Write the audio conversion code**\n\nThere's one wrinkle when connecting a phone call between Prism and Twilio's Media Streams: they don't speak the same audio format.\n\n-  [Twilio Media Streams sends and receives](https://www.twilio.com/docs/voice/media-streams/websocket-messages#start-message) 8 kHz[G.711 μ-law](https://en.wikipedia.org/wiki/G.711) .\n-  [Prism takes 16 kHz PCM16 in and sends 24 kHz PCM16 out](https://docs.rime.ai/docs/prism/overview) . It doesn't accept or send μ-law.\n\nSo on the way in, we need to decode μ-law to PCM16, while Rime's SDK resamples it for Prism. On the way out, we need to drop from 24 kHz down to 8 kHz for Media Streams.\n\nOur algorithm is relatively simple: run the audio through a low-pass filter with a 3.6 kHz cutoff – which removes nearly everything at 4 kHz and above – then keep every third sample.\n\nPaste the following code in your `audio.js`:\n\nHere's what the functions do:\n\n- `mulawToPcm16` decodes μ-law to PCM16. This decoder comes from[Rime's docs](https://docs.rime.ai/docs/prism/live-conversations#phone-audio) .\n- `encodeMulaw` converts one PCM16 sample into a μ-law byte.\n- `FILTER` is a low-pass filter. It keeps the 300–3,400 Hz telephone voice band (from[ITU-T G.712](https://www.itu.int/rec/T-REC-G.712) ), cuts off at 3.6 kHz, and removes (nearly) everything at 4 kHz and above.\n- `createReplyEncoder` keeps every third sample, leaving us with 8 kHz μ-law that Twilio can play.\n\n###  [**Step 5: Write the code**](#toc-heading-d828c4c2-f648-4637-a622-60d595908d23)\n\n**Step 5: Write the code**\n\nHere is a brief overview of the main components that will make up the AI voice assistant system we’ll be building.\n\n-  **Twilio Voice and Media Streams** : Handles the phone call and streams audio via WebSockets.\n-  **Node.js Server** : Your code and orchestration: manages the webhook, handles media conversion, connects to Rime, and includes any needed business logic.\n-  **Rime Prism** : Provides the speech-to-speech AI responses.\n\nNow, let's walk through building each part of the application step by step.\n\n####  [**Step 5.1: Import dependencies, load our Environment Variable, and initialize Fastify**](#toc-heading-0cafb0f4-d2d3-4078-933b-882cbdcda995)\n\nWe start by importing required modules and loading the environment variables from our `.env` file.\n\nPaste the following code in your `index.js`:\n\n####  [**Step 5.2: Define some constants**](#toc-heading-70e4df45-57b0-4c3d-b9d2-c6ccd8ef83c7)\n\n**Step 5.2: Define some constants**\n\nNext, we define constants for the voice, the server port, the assistant's first line, and its instructions.\n\nHere's what you should paste next in your file:\n\n`VOICE` controls how the assistant will sound. Prism has  [four voices](https://docs.rime.ai/docs/prism/sessions#choose-a-voice): `astra` (the default), `aurora`, `brando`, and `wawona`.\n\n`PORT` controls which port your application will open. We'll discuss this a little more in the ngrok section below.\n\n`MARK_INTERVAL_MS` controls how often we ask Twilio to confirm what the caller has heard. We'll explain why that matters in Step 5.5.\n\n`OPENING` is your assistant's first spoken line.\n\n`INSTRUCTIONS` is the system prompt. You might think a speech-to-speech model wouldn't need rules about markdown – but Prism speaks every word it writes, and without guidance, it can answer like a chat assistant, with lists and math notation. Rime recommends starting your instructions with rules for speech, so we do. \n\n *(Song lyrics without a song name? Wonder if you can find those anywhere else. And* [Alexander Graham Bell wanted everyone to say \"ahoy\"](https://www.twilio.com/docs/glossary/what-is-ahoy)  *–*  *so does your voice agent.)*\n\n####  [**Step 5.3: Define a tool**](#toc-heading-f0fb2f9b-5fb7-49d3-9740-ee81ddb82404)\n\n**Step 5.3: Define a tool**\n\nPrism can ask your backend to run a  *tool*, sometimes known as  *function calling*. To show a full tool round, we'll give the assistant one simple tool: the current time.\n\nPaste this next:\n\n`TIME_TOOL` is the definition for Prism: a name, a description, and a JSON Schema for the arguments. The  *description* is how Prism decides whether to call the tool – you need to write it the way you'd write documentation for a colleague who can't read your code.\n\n`getCurrentTime` is our tool. It's a mock, of course, and straightforward to demo – swap in your own business logic here later!\n\n####  [**Step 5.4: Define two routes**](#toc-heading-002b9f50-f747-4f5c-b126-5cd3255dff08)\n\n**Step 5.4: Define two routes**\n\nNow, let's get to the heart of the code. We create our Rime client, then define a root route (more of a health check…) and a route to handle incoming calls. The `/incoming-call` route will [return TwiML, Twilio's Markup Language](https://www.twilio.com/docs/voice/twiml), to direct Twilio how to handle an incoming call.\n\nPaste this into `index.js` after the tool:\n\nOur TwiML is short: a [`<Connect>`](https://www.twilio.com/docs/voice/twiml/connect) with a [`<Stream>`](https://www.twilio.com/docs/voice/twiml/stream). This opens a [bidirectional Media Stream](https://www.twilio.com/docs/voice/media-streams) to our `/media-stream` route. We don't need a `<Say>` greeting this time, since Prism greets the caller.\n\n`PUBLIC_HOST` is optional here, because some tunnels rewrite the `Host` header. If you use one of those, set `PUBLIC_HOST` in your `.env` to your tunnel's host name. Otherwise, the caller will hear dead air.\n\n####  [**Step 5.5: Connect the call audio to Prism, and allow us to interrupt**](#toc-heading-b93ea609-e39a-4230-801a-e723c4555270)\n\nThis is the biggest step, so it's reproduced here in two pieces. Rime's SDK runs the conversation with a function called `runConversation`. We give the SDK two adapters: a  *source* to read the caller's audio from, and  *playback* to send Prism's replies to.\n\nFirst, the source. Paste this next:\n\nTwilio pushes the caller's audio to us as fast as it can deliver it. Rime’s SDK reads it when it's ready.\n\nNext, the playback. This one's a bit longer – we'll explain it after the code block:\n\nPlaying audio is the most straightforward part: `write` converts a chunk (with our encoder from `audio.js`) and sends it to Twilio as a [` media` message](https://www.twilio.com/docs/voice/media-streams/websocket-messages#send-a-media-message).\n\n##### [Step 5.5.1: An interruption](#toc-heading-8a4bfcb9-e5e3-4847-a845-0835081d5f3e)\n\nLet us interrupt for a second to point out a tricky part of the code – uhh,  *interruptions*. The interesting part when handling interruptions is  *knowing what the caller heard*. When a caller talks over a reply, Prism wants to know how much of that reply they heard. But Twilio doesn't report where playback is, it just faithfully receives and sends audio chunks.\n\nSo, to handle it, we use [Media Streams `mark` messages](https://www.twilio.com/docs/voice/media-streams/websocket-messages#send-a-mark-message). After each 100 ms of audio, we send a reply mark and how many milliseconds we've sent so far... for example, `3:1200`. Twilio handles the marks and audio  *serially*, that is, it sends each mark back after the audio before it has played. Using the returned marks, we can estimate how much the caller heard (I mean, within the bounds of some additional latency and network effects. And, you know,  *also* within 100ms.).\n\nHere's what each method does:\n\n- `open` : Prism started a reply. We set up its bookkeeping and return an object for the SDK to write the reply's audio.\n- `write` : Converts the audio, sends it to Twilio in 100 ms pieces, and sends a mark after each piece.\n- `end` : Prism sent the last of this reply. We send a final`end` mark.\n- `completion` : Resolves with our estimate of*how much* of the reply the caller heard.\n- `interrupt` : The caller talked over a reply ('interrupted'). We send Twilio a[`clear` message](https://www.twilio.com/docs/voice/media-streams/websocket-messages#send-a-clear-message) to empty its buffer, then report how much the caller heard.\n- `stop` : The conversation ended, so we resolve any reply that's still playing.\n- `onMark` : Twilio played everything up to this mark. We update how much the caller heard. (After a`clear` , Twilio also returns any marks it was still holding. By then, we've already finished that reply, so we ignore them.)\n\n####  [**Step 5.6: Run tools and log the conversation**](#toc-heading-123250b9-cfa2-4683-a285-9e837c2e9bb4)\n\n**Step 5.6: Run tools and log the conversation**\n\nNext, we need to handle Prism's events. Paste this next:\n\n`onEvent` logs to your console: you will see when the caller starts speaking, what the caller said, and what the assistant said. The SDK calls this function for every event, and it must be synchronous – so we don't wait for tools here.\n\nWhen Prism asks for a tool, `runTool` runs it. It then gives Prism the result with `submitToolResult`, and asks Prism to continue its reply with `continueReply`. If the caller talks during a tool call, Prism answers the caller instead and drops the tool round – so `continueReply` throws, and we log it.\n\n####  [**Step 5.7: Open a Prism session for each call**](#toc-heading-b0f60691-612c-450f-963d-43c91aaabf04)\n\n**Step 5.7: Open a Prism session for each call**\n\nNow, we tie everything together. Paste this next:\n\n`converse` opens one Prism session for the call with our choice of voice (hey, that Rimes rhymes!), instructions, and tool. It logs the session ID – keep it handy if you ever need Rime's help with a call. Then it asks Prism to greet the caller with `requestReply`, and hands off to `runConversation`.\n\n####  [**Step 5.8: Handle the Media Stream**](#toc-heading-86c21f1c-74ef-40a4-8b3f-d8beb17cb936)\n\n**Step 5.8: Handle the Media Stream**\n\nTime to switch sides! Next, we set up the WebSocket route for the Media Stream – the route we give Twilio in our TwiML.\n\nPaste this next:\n\nHere's how we handle each Twilio message:\n\n- `start` : The stream has started. We create the playback for this call's stream (`createTwilioPlayback` ) and start the conversation. When the Prism session ends, we close the stream, and Twilio hangs up.\n- `media` : The caller's audio. We decode it and pass it to Prism (including silence, which Prism listens for to estimate when the caller is done).\n- `mark` : Twilio played a mark we sent, so pass it to the playback.\n- When the WebSocket closes, tell the SDK the call ended.\n- We also have two safety checks: a message that isn't JSON is logged and skipped. Also, a second `start` is ignored (so one stream never opens two Prism sessions).\n\n####  [**Step 5.9: Prepare the server**](#toc-heading-40cc334c-3df8-4ca4-bae1-44bc1dc773d8)\n\n**Step 5.9: Prepare the server**\n\nAnd finally, we start the Fastify server and bring it all home (using the port we passed in or defined). Paste this below what you have, and we're ready to roll! Or, you know, dial!\n\n####  [**Run the server**](#toc-heading-3374052f-4d59-4b2a-a731-b02b3aeb7105)\n\n**Run the server**\n\nCan't you hear the server saying  *\"Start Me Up?\"* – you can now run the server with the following command:\n\nIf the server starts successfully, you should see the message `Server is listening on port 5050` (or, alternatively, the port you specified) in your terminal.\n\n### [Step 6. Finish the basic integration](#toc-heading-16febb4d-47c2-445b-9401-684b00afdf17)\n\nNow it's time to provide instructions to Twilio and finish the wiring! We'll next cover using the reverse proxy `ngrok` to make your server publicly accessible.\n\n#### [**Step 6.1: Expose your server using ngrok**](#toc-heading-197d0346-8a4c-41a5-9835-ef9704d7c51f)\n\n**Step 6.1: Expose your server using ngrok**\n\nNow, you need to use ngrok or a similar service to expose your local server to the public internet. Twilio requires a public URL to send requests to your server and receive instructions from you.\n\nWe'll provide instructions for ngrok. You can [find other reverse proxy or tunneling options here](https://www.twilio.com/en-us/blog/expose-localhost-to-internet-with-tunnel).\n\nDownload and install ngrok if you still need to, then run the following command. And if you have changed the port from 5050, remember to update it here:\n\nThis will give you a public URL (e.g., `https://abc123.ngrok.app`) that you can use for testing.\n\n#### [**Step 6.2: Configure Twilio**](#toc-heading-b01cf8d1-9a47-4d14-be29-f2dda2e8d745)\n\n**Step 6.2: Configure Twilio**\n\nWe're so close now, *can you feel it in the air tonight?*\n\nGo to the [Twilio Console](https://1console.twilio.com/) and select the Voice-enabled number you want to call from **Numbers & senders** -> **Voice and emergency address**.\n\nSelect **Edit configuration details** and select the country, set the primary method to Use Webhooks and set your webhook URL to your ngrok URL, appending `/incoming-call`. For example, `https://abc123.ngrok.app/incoming-call`.\n\nSave your changes – *it's the final countdown*!\n\n#### [Step 6.3: Test your setup!](#toc-heading-d3aa128c-8a0f-47b5-b961-c92e05f689bb)\n\nMake sure your ngrok session is still running and your server is up. If it is, it's time – you can now make a call to your Twilio number using a cell phone or landline.\n\nThe assistant should greet you with \"Ahoy!\" Then, try a few things:\n\n- Greet it with *\"Hello? Is it me you're looking for?\"* .\n- Ask it to tell you a joke.\n- Ask what time it is somewhere, to hear and see a tool call.\n- Ask it to count to ten, then interrupt it partway through. Ask how far it got and see how close it matched your experience.\n\nAnd with that, you have a basic build of a Twilio Voice and Rime Prism AI Agent 🔥!\n\nPretty cool, but you can take it a step further – next, we’ll show you how you can persist customer details between calls, and analyze transcripts after a caller hangs up.\n\n### [Step 7 (optional): Personalize with Twilio Conversations](#toc-heading-3157ef9c-2e43-448b-937c-820e18fa0d54)\n\nYour assistant can hold a conversation – but once the caller hangs up and calls again, your current assistant build starts from scratch.\n\nIn this optional step, we will add intelligence and memory with two of [Twilio's Conversations products](https://www.twilio.com/docs/conversations).\n\n1. [Conversation Intelligence](https://www.twilio.com/docs/conversations/intelligence) runs operators on a transcript of each call when it ends.*Operators* are AI prompts that Twilio manages. We'll use two of them: a summary and a sentiment score.\n2. [Conversation Memory](https://www.twilio.com/docs/conversations/memory) turns those calls into notes about the caller, so your assistant can use them the next time the same number calls.\n\n#### [Step 7.1: Set up Conversations in the Console](#toc-heading-f2539518-850e-46de-8b5f-2f0d8681efec)\n\nYou'll create three resources:\n\n1. A [Memory Store](https://www.twilio.com/docs/conversations/memory/getting-started#set-up-your-memory-store) , which holds a profile for each caller.\n2. An [Intelligence Configuration](https://www.twilio.com/docs/conversations/intelligence/create-intelligence-configuration) with a rule that when a conversation ends (`CONVERSATION_END` ), run Twilio's ready-made Summary and Sentiment operators.\n3. A [Conversation Configuration](https://www.twilio.com/docs/conversations/orchestrator/concepts/core#conversation-configuration) that links the two. You’ll turn on`memoryExtractionEnabled` , which tells Memory to take notes from each call. You will set the voice channel's`statusTimeouts` to`null` so a conversation closes when the call ends.\n\nOnce you've created all three, you need to create a Twilio API key and secret ( [here's how to create one](https://www.twilio.com/docs/iam/api-keys/keys-in-console)). Add the above IDs to your `.env`, along with your key and secret.\n\n#### [Step 7.2: Transcribe the call](#toc-heading-ff578e6d-83b0-4bce-aa8d-27b2695d0290)\n\nFirst, we'll ask Twilio to transcribe the call and file it under your *Conversation Configuration*. Add this constant after `OPENING` in `index.js`:\n\nThen replace your `/incoming-call` route with this version:\n\nFour things changed in the TwiML:\n\n1. `<Start><Transcription>` transcribes the caller and the assistant.`conversationConfiguration` names the Conversation Configuration – that's what runs your operators and feeds Memory.\n2. Without `inboundTrackLabel=\"customer\"` , Twilio can't tell which side is the caller. Since we define it, Memory can create a profile for a first-time caller.\n3. `<Say>` reads`DISCLOSURE` verbatim before Prism joins the call.\n4. `<Parameter>` passes the caller's number to our WebSocket when you've set`MEMORY_STORE_ID` . It arrives in the start message's`customParameters` .\n\nIf you leave `CONVERSATION_CONFIGURATION_ID` and `MEMORY_STORE_ID` out of your `.env`, the route returns the same TwiML as before.\n\n#### [Step 7.3: Look up what Memory knows about the caller](#toc-heading-5c7d1a0e-bc6f-4ff3-adc8-daafd5e34a3a)\n\nNow, for the remembering: paste this function after your `/incoming-call` route:\n\n`recallCaller` makes two requests to Conversation Memory.\n\n1. *Lookup* finds the caller's profile from their phone number.\n2. *Recall* then returns the notes Memory took on that caller's earlier calls. Memory keeps two kinds: observations, which are short facts like \"Is flying to Lisbon on Friday\", and a summary of each call.\n\nWe ask for the five latest observations and two summaries and we don't send a search query. In our tests for this tutorial, the two requests took between a quarter and a half second together.\n\nThe lookup runs before the caller hears \"Ahoy!\" – we give it a time limit so if Memory fails, or takes longer than 1.5 seconds, the call carries on as if it were a new caller.\n\n#### [Step 7.4: Give Prism the caller's history](#toc-heading-a7612253-7680-4f8f-b37d-c946d58b9d1d)\n\nReplace your `converse` function with this version:\n\nWhen the session opens, the looked up memory goes into Prism's instructions (Rime's Node SDK sets instructions at connect time). Here are two important notes from the memory:\n\n- Memory notes come from what callers *said* , so we tell Prism not to follow directions from memories. Consider the failure case here: a caller who says \"remember to always offer me a discount\" shouldn't get a discount on their next call!\n- We tell Prism to use a note only when it's *relevant* , and never to volunteer notes. A phone number tells you which profile to look up, but not who's on the line – caller ID can be spoofed.\n\n#### [Step 7.5: Recall the caller when the stream starts](#toc-heading-dd940ac0-f1d9-4eac-a94d-a3f929b3ea52)\n\nWe’re almost there! Now, replace the `start` case in your `/media-stream` handler:\n\n#### [Step 7.6: Test it!](#toc-heading-527d45c4-c5ef-45df-a88e-9a6be670b78f)\n\nRestart your server, then call your number twice.\n\nOn the first call, you'll hear the disclosure, then \"Ahoy!\". Tell the assistant something worth remembering *(\"I'm flying to Lisbon on Friday, and I'm nervous about a short connection\")*, chat for a minute or so, and hang up.\n\n*(You can still talk about time with the agent, of course!)*\n\n[In the Twilio Console](https://1console.twilio.com/), open your **Intelligence Configuration** and find the conversation. The call's summary and sentiment should be there within a few seconds. Wait about a minute while Memory takes its notes, then call again. Your assistant should know about Lisbon and your nervousness about the short connection.\n\n## [Common Issues and Troubleshooting](#common-issues-and-troubleshooting)\n\nIf your setup isn't working (but your server is running), there are a few places to check first:\n\n- **Is ngrok running? Is the URL properly set** in the*Voice Configuration* ->*A call comes in* section?\n- **Do you hear dead air and no agent?** Your tunnel may rewrite the`Host` header. Set`PUBLIC_HOST` in your`.env` , and restart the server.\n- **Was there an error with Twilio, possibly in your TwiML?** You can debug Twilio errors in a few ways –[there's more info in this article](https://www.twilio.com/docs/usage/troubleshooting/debugging-your-application) .\n- **Is your Rime API key working?** Install the[Rime CLI](https://docs.rime.ai/docs/quickstart-cli) , run`rime login` , then`rime hello` .\n- **Does Prism report an error?** Your console logs each Prism error with its code. Rime's[errors and limits docs](https://docs.rime.ai/docs/prism/errors#error-codes) explain the codes.\n- **Does the assistant reply too early, or does it cut you off?** Prism might take a long pause as the end of a turn – tune that with the[`turnDetection`](https://docs.rime.ai/docs/prism/sessions#configure-a-session) option when you connect.\n- **Did the call end after a quiet stretch?** Prism closes a conversation[after five minutes](https://docs.rime.ai/docs/prism/sessions#end-a-session) with no speech and no reply. Streaming silence doesn't keep it open.\n- **Does your assistant remember nothing on the second call?** Check that memoryExtractionEnabled is on in your Conversation Configuration, it's off by default.\n\n## [Conclusion](#toc-heading-a1318baa-5bb7-42b9-82cb-0ff299be7aef)\n\nIsn't that amazing? You've now successfully built an AI voice assistant using Twilio Voice and Rime's Prism. Now you can *call me, call me any, anytime.* (Well, if you name your agent ‘me’, anyway.)\n\nCongratulations to Rime on Prism – we’re looking forward to seeing what you build with Rime Prism and Twilio. And *we can’t wait to make owl jokes and quote song lyrics to what you build!*\n\nNext steps:\n\n- Check out the [Twilio Media Streams documentation](https://www.twilio.com/docs/voice/media-streams) and[Media Streams WebSocket messages](https://www.twilio.com/docs/voice/media-streams/websocket-messages) for more on`media` ,`mark` , and`clear` .\n- See [Rime's Prism documentation](https://docs.rime.ai/docs/prism/overview) for more advanced features.\n- Read [Core Latency in AI Voice Agents](https://www.twilio.com/en-us/blog/developers/best-practices/guide-core-latency-ai-voice-agents) for more on where the milliseconds go in a voice agent.\n\n*Paul Kamp is the Technical Editor-in-Chief of the Twilio Blog. Reach him at pkamp [at] twilio.com, (not 867-5309).*\n\n*Safdar Jaffari is a Principal Partner Engineer at Twilio, working with partners on data, communications and AI initiatives. He can be reached at sjaffari [at] twilio.com.* \n\n## Related Posts\n\n## Related Resources\n\nTwilio Docs\n\nFrom APIs to SDKs to sample apps\n\nAPI reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.\n\nResource Center\n\nThe latest ebooks, industry reports, and webinars\n\nLearn from customer engagement experts to improve your own communication.\n\nAhoy\n\nTwilio's developer community hub\n\nBest practices, code samples, and inspiration to build communications and digital engagement experiences.", "url": "https://wpnews.pro/news/build-an-ai-voice-assistant-with-twilio-voice-and-media-streams-rime-s-prism-and", "canonical_source": "https://www.twilio.com/en-us/blog/developers/tutorials/integrations/voice-ai-agent-rime-prism-node", "published_at": "2026-10-30 00:00:00+00:00", "updated_at": "2026-10-30 22:47:41.561921+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "generative-ai"], "entities": ["Twilio", "Rime", "Prism", "Twilio Media Streams", "Node.js", "Twilio Conversation Memory", "Twilio Conversation Intelligence", "Fastify"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/build-an-ai-voice-assistant-with-twilio-voice-and-media-streams-rime-s-prism-and", "markdown": "https://wpnews.pro/news/build-an-ai-voice-assistant-with-twilio-voice-and-media-streams-rime-s-prism-and.md", "text": "https://wpnews.pro/news/build-an-ai-voice-assistant-with-twilio-voice-and-media-streams-rime-s-prism-and.txt", "jsonld": "https://wpnews.pro/news/build-an-ai-voice-assistant-with-twilio-voice-and-media-streams-rime-s-prism-and.jsonld"}}