How I Built Maya: A Real-Time Voice AI Clinic Receptionist

A developer built Maya, a real-time voice AI receptionist for a fictional Bengaluru clinic, using LiveKit, Groq, and Cartesia to handle appointment bookings with sub-second latency. The project showcases a full voice pipeline and addresses challenges like rate limits by compressing prompts to cut token usage by 73%.

Most AI demos you see online are either simple text chatbots or basic wrappers around an API. I wanted to build something that felt like a real product solving a real problem. So I built Maya—a real-time voice AI receptionist for a fictional clinic in Bengaluru called SwasthyaCare Clinic. Instead of typing into a chat box, you just talk to your microphone like you're on a real phone call. Maya answers your questions, checks doctor schedules, books appointments, reschedules or cancels existing visits, and gives clinic information—all in real-time with sub-second voice latency. Here is a breakdown of how the architecture works, the real issues I faced during development rate limits, background noise, Docker container paths , and how I solved them. When someone calls a clinic, they don't want to navigate a robotic IVR menu "Press 1 for appointments..." . They want to talk to a human receptionist. Maya handles that conversational flow: • Checks live doctor availability: Understands dates and doctor specialties General Physician vs. Dentist . • Gathers details naturally: Asks for missing information name, 10-digit mobile number, preferred slot across multiple turns. • Explicit confirmation before booking: Summarizes the appointment details and only commits the booking once the user says "Yes". • Enforces business rules: Only books up to 14 days in advance, checks for slot collisions, and enforces a strict 2-hour cancellation/rescheduling policy. • Medical triage guardrails: If someone mentions severe symptoms like acute chest pain or breathing issues, Maya immediately instructs them to dial 112 or visit an emergency room rather than booking a routine outpatient slot. To make voice feel natural, latency has to be as close to human conversational speed as possible. A typical turn looks like this: User speaks into Browser Mic ↓ WebRTC Audio Stream LiveKit Cloud ap-south / Mumbai ↓ Silero VAD Voice Activity Detection on-device ↓ Groq Whisper Speech-to-Text ↓ Groq LLM Reasoning + Function/Tool Calling ↓ SQLite Database Appointment State Engine ↓ Cartesia TTS Streaming Indian English Voice ↓ WebRTC Audio Browser Speakers User hears response • Audio Transport: LiveKit WebRTC . Handles real-time, low-latency audio streaming between the browser and backend worker. • VAD Voice Activity Detection : Silero VAD. Runs locally inside the worker to detect when the user starts and stops speaking. • STT Speech-to-Text : Groq Whisper whisper-large-v3-turbo . Transcribes speech into text in ~100–200ms. • LLM Reasoning & Tool Calling: Groq openai/gpt-oss-120b . Fast reasoning and function calling. • TTS Text-to-Speech : Cartesia sonic-turbo . Uses the "Priya" voice profile—a natural, clear Indian English tone suited for a Bengaluru clinic. • State & Database: SQLite + Python. Manages appointments, availability checks, and audit trails. • Frontend: React + Vite with @livekit/components-react and Tailwind CSS. Building the basic happy path is easy; making real-time voice work reliably is where the real learning happened. Here are 4 specific problems I ran into: When I first ran voice tests, everything would work for 2 or 3 turns, and then suddenly crash with an HTTP 429 Too Many Requests Rate Limit Exceeded . Why it happened: On Groq's free tier, there is an 8,000 Tokens Per Minute TPM limit. My initial system prompt combined with the JSON schemas for 5 function tools was taking ~1,900 tokens per single LLM call. If the user spoke 4 times in a minute, that was 1,900 4 = 7,600+ tokens, immediately blowing through the 8,000 token limit. How I fixed it: I completely compressed the system prompt and tool docstrings. I removed repetitive instructions, used dense bullet points, and kept the tool parameters minimal while preserving all clinical guardrails. This brought the request size down from ~1,900 tokens to ~500 tokens a 73% reduction . Suddenly, I could have 15+ turns a minute without hitting rate limits. While testing with my laptop mic, I noticed the agent would randomly trigger and start speaking even when I hadn't said anything. Why it happened: Default VAD sensitivity was picking up subtle background sounds—fan noise, keyboard typing, and breathing. Each micro-sound triggered LiveKit's turn detector, which immediately dispatched an STT call, an LLM call, and a TTS synthesis call. This was burning API quota and interrupting the conversation. How I fixed it: I tuned the Silero VAD parameters and turn handling in LiveKit: vad instance = silero.VAD.load min speech duration=0.25, Ignore clicks and breath sounds under 250ms min silence duration=0.65, Wait for a clean pause before marking end-of-turn prefix padding duration=0.3, activation threshold=0.6, Require clearer vocal energy over ambient noise session = AgentSession turn handling={ "endpointing": {"min delay": 0.6, "max delay": 3.0}, "preemptive generation": {"enabled": False}, Only generate audio when speech is complete "interruption": {"enabled": True, "min duration": 0.5}, } Disabling preemptive generation and requiring at least 250ms of vocal energy eliminated the false triggers completely. Initially, I used Groq's built-in TTS. While it worked, it only had US and Arabic voice profiles, and the free-tier rate limits were strict for voice generation. For a clinic located in HSR Layout, Bengaluru, a North American voice felt out of place. I integrated Cartesia's sonic-turbo model with their Priya voice an Indian English female voice profile . The difference was night and day: • Latency dropped below 100ms. • The cadence and pronunciation of Indian names sounded authentic. When I tried deploying the Python backend, I quickly realized you can't just throw a voice agent onto serverless platforms like Vercel or AWS Lambda. A text chatbot handles a quick HTTP request and terminates in 1 second. A voice agent, on the other hand, is a persistent WebRTC worker daemon. It maintains an active bidirectional audio socket to LiveKit Cloud 24/7. The Solution: • React Frontend: Deployed on Vercel with a serverless token endpoint /api/token that generates short-lived LiveKit JWT access tokens without exposing LIVEKIT API SECRET to the browser. • Python Agent Worker: Containerized via Docker and deployed to LiveKit Cloud Agent Hosting in the ap-south Mumbai region for ultra-low ping. When deploying the Docker container, I ran into a ModuleNotFoundError: No module named 'agent.prompts'. The container entrypoint was running python agent/agent.py start, which put /app/agent into Python's sys.path instead of the root /app. I fixed this by adding ENV PYTHONPATH="/app" in the Dockerfile and adding a defensive path resolution in Python. This project gave me a massive appreciation for what it takes to build reliable real-time AI systems. I'd love to connect with other engineers and builders working in Voice AI, WebRTC, and LLM tool calling: • How are you handling ambient noise and interruption handling in your voice agents? • What TTS providers have you found best for regional accents? Check out the code and feel free to share your thoughts or suggestions 🔗 GitHub Repo: https://github.com/CosmosTechy/maya-ai-voice-receptionist https://github.com/CosmosTechy/maya-ai-voice-receptionist 🌐 Live Demo: https://maya-ai-voice-receptionist.vercel.app/ https://maya-ai-voice-receptionist.vercel.app/