Add Token Streaming and Interruption Handling to a Twilio Voice Anthropic Integration

Twilio published a follow-up developer tutorial showing how to add token streaming and interruption-aware conversation turn tracking to a Node.js integration between Twilio Voice's Conversation Relay and Anthropic's Claude models, addressing the latency and interruption weaknesses of the earlier basic app. The tutorial streams partial Claude responses to Conversation Relay as text tokens with the attribute last: true so callers hear the AI's reply before it finishes generating, and the code is available in the demo repository's step2-streaming-tokens and step3-conversation-tracking branches. The setup requires Node.js, a Twilio phone number, Ngrok, and an Anthropic API key.

Add Token Streaming and Interruption Handling to a Twilio Voice Anthropic Integration Time to read: Written by Reviewed by Conversation Relay https://www.twilio.com/docs/voice/twiml/connect/conversationrelay from Twilio lets you build real time, human-like voice applications with AI Large Language Models of your choice. It opens a WebSocket with your app so you can integrate your choice of AI API with Twilio Voice https://www.twilio.com/en-us/voice . In my previous post, I showed you one way to integrate Anthropic with Twilio Voice using Conversation Relay in Node.js. This setup allowed you to create a basic real-time voice assistant using Anthropic's Claude models https://www.anthropic.com/claude . However, there were some weaknesses to that basic app, particularly around latency and handling interruptions during conversations. In this post, I'll show you how to implement token streaming to reduce voice latency and add local tracking for interruption-aware conversation turn handling. Let’s dive in Prerequisites toc-heading-bc5bb598-5944-44c3-854b-004b940c52be Prerequisites Before you get started, you’ll need a few accounts and a few things ready : - Node.js https://nodejs.org/en installed - A Twilio phone number Sign up for Twilio https://console.twilio.com/ - An IDE or code editor - Ngrok https://ngrok.com/ to expose your server - An Anthropic Account https://platform.claude.com/login and API Key - A phone to place a phone call to test the integration And with those things in hand, let’s get started… Add token streaming toc-heading-087b8c30-ddf2-44dc-a83d-4b69789486c9 Add token streaming The first thing you are going to address from the basic app is the latency https://www.twilio.com/docs/glossary/what-is-latency . If you prompt the basic app with something requiring a longer response – for example, “name ten things that happened in 1999” – you’ll notice it takes a few seconds before you hear the AI’s response. Some of that pause is because we’re waiting on the full response from the AI and you guaranteed a long one with a prompt like that . You're going to address some of the latency by streaming tokens from Anthropic’s Claude instead of waiting on a complete response. What is Token Streaming? toc-heading-f62bdfe0-9c58-4892-80c1-1e68cecce112 Tokens are the smallest units of a response generated by an AI Language Model https://platform.claude.com/docs/en/about-claude/glossary tokens . In your basic app, you were waiting for the entire response – including all the tokens – before we sent the AI’s text to our text to speech providers. But waiting for a whole response isn’t the only mode supported by Anthropic and other providers. You can also receive partial responses https://platform.claude.com/docs/en/build-with-claude/streaming from a model and incrementally stream tokens to your app. From there, you can send them along to Conversation Relay as text tokens with the attribute last: true https://www.twilio.com/docs/voice/twiml/connect/conversationrelay text-tokens-message , and a user will hear the AI’s response before it’s even finished coming up with it . Modify how the AI responds modify-how-the-ai-responds Our demo repository has branches showing every step we took to enhance the base tutorial. If you get stuck in this post, you can clone Step Two: Streaming Tokens https://github.com/pkamp3/cr-anthropic-demo/tree/step2-streaming-tokens and Step Three: Conversation Tracking https://github.com/pkamp3/cr-anthropic-demo/tree/step3-conversation-tracking from our repo. This follow-up blog post discusses Tool/Function Calling https://www.twilio.com/en-us/blog/developers/tutorials/product/function-calling-twilio-voice-anthropic-claude-integration . You need to make two changes to our code to add token streaming. You'll first change how you call Anthropic and wait for a response, then modify how you handle the Conversation Relay ’prompt’ message type. Change the aiResponse function toc-heading-02542e92-6414-48d3-b6bc-f86396f9453f aiResponse function Open your server.js and locate the aiResponse function. You’re going to change its name to aiResponseStream – since that’s now more descriptive – and write a little code to handle the streamed tokens. Here’s the code before: Once you’ve located that block, change the code to this: When you use the Anthropic library’s stream function, Anthropic changes to token streaming mode and sends one token at a time. Your app then forwards tokens to Conversation Relay with last: false so Conversation Relay knows to expect more. And finally, when the response is done, this code sends an empty text message with last: true and returns the final accumulated response from the assistant using stream.finalMessage . Change how you handle the Conversation Relay ‘prompt’ message toc-heading-b2bc0ed4-fa5a-4cef-bc27-6e63e944f950 In server.js , locate the case "prompt": in your switch block in fastify.register . Since you’re now calling an asynchronous function that returns a Promise https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global Objects/Promise , you need to wait for the Promise to return before adding the “assistant” response to the conversation. Here’s the code before: Change that code to the following: As you can see, you now await the return from the aiResponseStream function, and push the role and content from the LLM into the local conversation storage. Add conversation tracking and interruption handling toc-heading-b2025b0d-f9a5-4822-bcbd-de6ed38e9548 Nice work so far If you test the app again – and try your open-ended prompt of “name ten things that happened in 1999” – you should notice much improved latency. But if you try to interrupt the AI you might notice another problem: while it sounds verbally like you interrupted its response, the AI doesn’t know how much of the response you heard In this step, you’ll track the conversation locally, and show one way to handle these interruptions and keep your AI informed using Conversation Relay’s utteranceUntilInterrupt attribute https://www.twilio.com/docs/voice/conversationrelay/websocket-messages interrupt-message on interrupt messages and indexing into local conversation storage. Modify the AI response function again modify-the-ai-response-function-again Head back to your aiResponseStream function. You’re going to update it further to accumulate tokens as they arrive: You’re now both forwarding tokens to Conversation Relay and accumulating them in the assistantSegments array. When the AI’s response is done, you push the complete response into sessionData.conversation where we are storing all of the turns of the conversation. As you see in a minute, we will index into those conversation turns to determine where we interrupt. But first: more changes to the ’prompt’ message handler. Change how we manage the ‘prompt’ message toc-heading-aa55153d-e2b7-4a42-8327-0b0e34b7b033 You're yet again changing how you track the conversation. Since aiResponseStream now handles accumulating tokens and adding them to the session conversation history, you can simplify the ’prompt’ message case in your case block. Change the ’prompt’ case to: … and then move on to ‘interrupt’ messages. Handle ‘interrupt’ messages toc-heading-4fa279ec-6ed2-437e-9819-8ea1f04b9c41 Now, modify the WebSocket handling to better handle the interrupt message from Conversation Relay. Start by calling a new function, handleInterrupt , when we get the interrupt message from Conversation Relay: Now, turn your attention to this new function. It’ll take the utteranceUntilInterrupt from the interrupt message along with the callSid which the app is using to index the conversation turns. Here’s the function to add: As you saw, the handleInterrupt function is only called if a voice interaction is interrupted – that is, when Conversation Relay sends the interrupt message. This code finds the utteranceUntilInterrupt in your local conversation storage and replaces that part of the AI’s turn with what Conversation Relay told your app the user has heard. Then, the next time you pass the conversation to Anthropic with aiResponseStream conversation, ws , the AI will ‘know’ how far the caller heard. Pretty cool, right? If you missed a step and it isn’t working, you can see the complete code for this step, here https://github.com/pkamp3/cr-anthropic-demo/tree/step3-conversation-tracking . Test your AI application toc-heading-c4a94e4d-47b9-4457-8d40-8aaa6175bfd8 Test your AI application It’s time for the payoff; you’re ready to test You might already have these initial steps memorized or finished for the basic tutorial. Feel free to skip the setup if so. Set up the AI app set-up-the-ai-app In your terminal, open up a connection using ngrok: Now, update your .env with your new ngrok URL do not include the scheme the “https://” or “http://” : Now, start up the server: Go into your Twilio console and find the phone number to use https://console.twilio.com/us1/develop/phone-numbers/manage/incoming . Set the configuration under A call comes in with the Webhook option and HTTP GET in the HTTP field. For the URL , add your ngrok URL this time include the “https://” , and add /twiml . For example, https://abcdefgh.ngrok.app/twiml . Hit Save , you’re ready to test. Call Claude call-claude Dial that Twilio number you set up. You should soon hear a greeting – talk to your new, enhanced Claude – Anthropic’s AI Assistant Try interrupting the AI when it is responding to your prompts. If you’re sick of asking about 1999, ask about various dog breeds, or ask it to come up with a long, complicated sentence with a lot of commas that seems to run on and never end, despite its best intentions, even though long, complicated sentences aren’t usually appropriate for voice conversations. Whatever response you interrupt, test by asking how far the AI got before you interrupted. Next steps toc-heading-06a9493a-26d6-4243-a8c4-6163b01c0136 Next steps Congratulations on the new, more powerful AI voice assistant you just built with Node.js, Conversation Relay, Claude, and a few lines of code In the next post https://www.twilio.com/en-us/blog/developers/tutorials/product/function-calling-twilio-voice-anthropic-claude-integration , we’ll explore tool or function calling to expand your AI assistant’s horizons just a little bit further. Paul Kamp is the Technical Editor-in-Chief of the Twilio Blog. He felt awkward continually interrupting the AI. You can reach Paul – and don’t worry, you aren’t interrupting – at pkamp at twilio.com. Related Posts Related Resources Twilio Docs From APIs to SDKs to sample apps API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform. Resource Center The latest ebooks, industry reports, and webinars Learn from customer engagement experts to improve your own communication. Ahoy Twilio's developer community hub Best practices, code samples, and inspiration to build communications and digital engagement experiences.