Build a Real-Time Voice Translation App

Getting Started with a Real‑Time Voice Translation App

Building a real‑time voice translation app feels like science‑fiction, but with today’s cloud APIs it’s a very achievable side‑project. In this article we’ll walk through the whole stack: capturing audio, transcribing it, translating the text, and finally turning the translated text back into speech. We’ll keep the example language‑agnostic (Python on the backend, JavaScript on the frontend) and we’ll lean on a few battle‑tested services. The star of the show is ElevenLabs for its high‑quality text‑to‑speech (TTS) and voice‑cloning capabilities. You can start experimenting right away with their free tier at https://try.elevenlabs.io/kr07zfuqn1bp.

If you’re building a language‑learning app, a travel chatbot, or a customer‑support tool, latency is everything. A 2‑second delay feels natural; a 5‑second lag is noticeable. That means we need:

The rest of this guide assumes you already have an API key for a STT provider (Google Cloud Speech‑to‑Text, Azure Speech, or Whisper on your own server) and a translation service (Google Translate, DeepL, or Microsoft Translator). We’ll plug in ElevenLabs for the TTS step.

+----------------+      +-------------------+      +-----------------+
|  Browser (JS)  | <--->|  Node.js/Flask API| <--->|  ElevenLabs TTS |
+----------------+      +-------------------+      +-----------------+
        ^                        ^                          ^
        |                        |                          |
        |  WebSocket (audio)     |  REST (transcribe/translate)
        |                        |                          |
        |                        |                          |
  Mic input  <---  STT API  <---  Translation API  <---  TTS Engine

Below is a minimal JavaScript snippet that records mic input and streams it to a server using WebSocket. We’re using the MediaRecorder API for simplicity.

const socket = new WebSocket('wss://your-server.com/stream');
const mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });

mediaRecorder.ondataavailable = (e) => {
  // Send each chunk as binary data
  socket.send(e.data);
};

socket.addEventListener('message', (event) => {
  // event.data is the TTS audio blob from ElevenLabs
  const audio = new Audio(URL.createObjectURL(event.data));
  audio.play();
});

mediaRecorder.start(250); // send 250ms chunks

What to watch for

MediaRecorder. For older browsers you may need a polyfill or a server‑side WebRTC solution. We’ll use Python + FastAPI for the backend because of its async support and excellent typing. The core flow is:

from fastapi import FastAPI, WebSocket
import aiohttp
import asyncio
import json

app = FastAPI()

STT_ENDPOINT = "https://api.speech-to-text.com/v1/recognize"
TRANSLATE_ENDPOINT = "https://api.translate.com/v1/translate"
ELEVENLABS_API = "https://api.elevenlabs.io/v1/text-to-speech"

ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_KEY"

async def transcribe_audio(audio_bytes: bytes) -> str:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            STT_ENDPOINT,
            headers={"Authorization": f"Bearer {STT_ENDPOINT}"},
            data=audio_bytes,
        ) as resp:
            result = await resp.json()
            return result["text"]

async def translate_text(text: str, target_lang: str = "es") -> str:
    async with aiohttp.ClientSession() as session:
        payload = {"q": text, "target": target_lang}
        async with session.post(
            TRANSLATE_ENDPOINT,
            json=payload,
            headers={"Authorization": f"Bearer {TRANSLATE_ENDPOINT}"},
        ) as resp:
            result = await resp.json()
            return result["translatedText"]

async def synthesize_speech(text: str) -> bytes:
    async with aiohttp.ClientSession() as session:
        payload = {
            "text": text,
            "voice_settings": {"pitch": 0, "speed": 1}
        }
        async with session.post(
            ELEVENLABS_API,
            json=payload,
            headers={
                "xi-api-key": ELEVENLABS_API_KEY,
                "Content-Type": "application/json",
            },
        ) as resp:
            return await resp.content.read()

@app.websocket("/stream")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    buffer = bytearray()
    while True:
        data = await websocket.receive_bytes()
        buffer.extend(data)

        if len(buffer) > 300_000:  # roughly 3 s of 96 kHz audio
            text = await transcribe_audio(buffer)
            translated = await translate_text(text, target_lang="de")
            audio = await synthesize_speech(translated)
            await websocket.send_bytes(audio)
            buffer.clear()

ElevenLabs offers:

Tip: The free tier gives you a generous amount of usage per month. If you’re building a prototype, you’ll be fine. When you go live, just swap the API key in the ELEVENLABS_API_KEY variable.

target_lang parameter. With a handful of lines of code and a few paid services, you can turn a microphone into a real‑time, multilingual chatbot. The key is to keep the data path tight: mic → STT → translation → TTS → speaker. ElevenLabs makes the last hop the smoothest, delivering natural‑sounding voices that feel like a real person.

Ready to bring your voice translation idea to life? Sign up for ElevenLabs today and unlock high‑quality TTS and voice‑cloning for your app.

Try ElevenLabs now: https://try.elevenlabs.io/kr07zfuqn1bp

Happy coding!