Build a Real-Time Voice Translation App

A developer published a walkthrough for building a real-time voice translation app that chains speech-to-text, machine translation, and text-to-speech services, using ElevenLabs for voice synthesis. The guide pairs a JavaScript MediaRecorder frontend with a Python FastAPI backend that streams 250ms audio chunks over WebSocket, targeting roughly two seconds of end-to-end latency.

Getting Started with a Real‑Time Voice Translation App Building a real‑time voice translation app feels like science‑fiction, but with today’s cloud APIs it’s a very achievable side‑project. In this article we’ll walk through the whole stack: capturing audio, transcribing it, translating the text, and finally turning the translated text back into speech. We’ll keep the example language‑agnostic Python on the backend, JavaScript on the frontend and we’ll lean on a few battle‑tested services. The star of the show is ElevenLabs for its high‑quality text‑to‑speech TTS and voice‑cloning capabilities. You can start experimenting right away with their free tier at https://try.elevenlabs.io/kr07zfuqn1bp https://try.elevenlabs.io/kr07zfuqn1bp . If you’re building a language‑learning app, a travel chatbot, or a customer‑support tool, latency is everything. A 2‑second delay feels natural; a 5‑second lag is noticeable. That means we need: The rest of this guide assumes you already have an API key for a STT provider Google Cloud Speech‑to‑Text, Azure Speech, or Whisper on your own server and a translation service Google Translate, DeepL, or Microsoft Translator . We’ll plug in ElevenLabs for the TTS step. +----------------+ +-------------------+ +-----------------+ | Browser JS | <--- | Node.js/Flask API| <--- | ElevenLabs TTS | +----------------+ +-------------------+ +-----------------+ ^ ^ ^ | | | | WebSocket audio | REST transcribe/translate | | | | | | Mic input <--- STT API <--- Translation API <--- TTS Engine Below is a minimal JavaScript snippet that records mic input and streams it to a server using WebSocket. We’re using the MediaRecorder API for simplicity. js const socket = new WebSocket 'wss://your-server.com/stream' ; const mediaRecorder = new MediaRecorder stream, { mimeType: 'audio/webm' } ; mediaRecorder.ondataavailable = e = { // Send each chunk as binary data socket.send e.data ; }; socket.addEventListener 'message', event = { // event.data is the TTS audio blob from ElevenLabs const audio = new Audio URL.createObjectURL event.data ; audio.play ; } ; mediaRecorder.start 250 ; // send 250ms chunks What to watch for MediaRecorder . For older browsers you may need a polyfill or a server‑side WebRTC solution. We’ll use Python + FastAPI for the backend because of its async support and excellent typing. The core flow is: python main.py from fastapi import FastAPI, WebSocket import aiohttp import asyncio import json app = FastAPI Replace with your real keys STT ENDPOINT = "https://api.speech-to-text.com/v1/recognize" TRANSLATE ENDPOINT = "https://api.translate.com/v1/translate" ELEVENLABS API = "https://api.elevenlabs.io/v1/text-to-speech" ELEVENLABS API KEY = "YOUR ELEVENLABS KEY" async def transcribe audio audio bytes: bytes - str: async with aiohttp.ClientSession as session: async with session.post STT ENDPOINT, headers={"Authorization": f"Bearer {STT ENDPOINT}"}, data=audio bytes, as resp: result = await resp.json return result "text" async def translate text text: str, target lang: str = "es" - str: async with aiohttp.ClientSession as session: payload = {"q": text, "target": target lang} async with session.post TRANSLATE ENDPOINT, json=payload, headers={"Authorization": f"Bearer {TRANSLATE ENDPOINT}"}, as resp: result = await resp.json return result "translatedText" async def synthesize speech text: str - bytes: async with aiohttp.ClientSession as session: payload = { "text": text, "voice settings": {"pitch": 0, "speed": 1} } async with session.post ELEVENLABS API, json=payload, headers={ "xi-api-key": ELEVENLABS API KEY, "Content-Type": "application/json", }, as resp: ElevenLabs streams MP3 return await resp.content.read @app.websocket "/stream" async def websocket endpoint websocket: WebSocket : await websocket.accept buffer = bytearray while True: data = await websocket.receive bytes buffer.extend data Simple logic: every 3 seconds of audio we process if len buffer 300 000: roughly 3 s of 96 kHz audio text = await transcribe audio buffer translated = await translate text text, target lang="de" audio = await synthesize speech translated await websocket.send bytes audio buffer.clear ElevenLabs offers: Tip : The free tier gives you a generous amount of usage per month. If you’re building a prototype, you’ll be fine. When you go live, just swap the API key in the ELEVENLABS API KEY variable. | Step | What to do | Notes | |---|---|---| | 1 | Deploy the FastAPI app to a platform that supports WebSockets e.g., Fly.io, Render, Railway . | Make sure you set CORS ALLOW ORIGIN to your domain. | | 2 | Secure your API keys. Store them in environment variables or a secrets manager. | Don’t hard‑code them in the repo. | | 3 | Enable HTTPS on the front‑end. Browsers block microphone access on insecure origins. | Let’s Encrypt is free. | | 4 | Test latency. Use chrome://inspect or websocket.org to measure round‑trip time. | Aim for < 500 ms end‑to‑end. | | 5 | Optimize VAD. If you notice unnecessary STT calls, add a simple VAD library like webrtcvad . | Reduces costs and latency. | target lang parameter. With a handful of lines of code and a few paid services, you can turn a microphone into a real‑time, multilingual chatbot. The key is to keep the data path tight: mic → STT → translation → TTS → speaker. ElevenLabs makes the last hop the smoothest, delivering natural‑sounding voices that feel like a real person. Ready to bring your voice translation idea to life? Sign up for ElevenLabs today and unlock high‑quality TTS and voice‑cloning for your app. Try ElevenLabs now : https://try.elevenlabs.io/kr07zfuqn1bp https://try.elevenlabs.io/kr07zfuqn1bp Happy coding