# How I Built Maya: A Real-Time Voice AI Clinic Receptionist

> Source: <https://dev.to/cosmostechy/how-i-built-maya-a-real-time-voice-ai-clinic-receptionist-45m>
> Published: 2026-09-04 08:12:17+00:00

Most AI demos you see online are either simple text chatbots or basic wrappers around an API.

I wanted to build something that felt like a real product solving a real problem. So I built Maya—a real-time voice AI receptionist for a fictional clinic in Bengaluru called SwasthyaCare Clinic.

Instead of typing into a chat box, you just talk to your microphone like you're on a real phone call. Maya answers your questions, checks doctor schedules, books appointments, reschedules or cancels existing visits, and gives clinic

information—all in real-time with sub-second voice latency.

Here is a breakdown of how the architecture works, the real issues I faced during development (rate limits, background noise, Docker container paths), and how I solved them.

When someone calls a clinic, they don't want to navigate a robotic IVR menu ("Press 1 for appointments..."). They want to talk to a human receptionist.

Maya handles that conversational flow:

• Checks live doctor availability: Understands dates and doctor specialties (General Physician vs. Dentist).

• Gathers details naturally: Asks for missing information (name, 10-digit mobile number, preferred slot) across multiple turns.

• Explicit confirmation before booking: Summarizes the appointment details and only commits the booking once the user says "Yes".

• Enforces business rules: Only books up to 14 days in advance, checks for slot collisions, and enforces a strict 2-hour cancellation/rescheduling policy.

• Medical triage guardrails: If someone mentions severe symptoms like acute chest pain or breathing issues, Maya immediately instructs them to dial 112 or visit an emergency room rather than booking a routine outpatient slot.

To make voice feel natural, latency has to be as close to human conversational speed as possible. A typical turn looks like this:

```
User speaks into Browser Mic
       ↓ (WebRTC Audio Stream)
LiveKit Cloud (ap-south / Mumbai)
       ↓
Silero VAD (Voice Activity Detection on-device)
       ↓
Groq Whisper (Speech-to-Text)
       ↓
Groq LLM (Reasoning + Function/Tool Calling)
       ↓
SQLite Database (Appointment State Engine)
       ↓
Cartesia TTS (Streaming Indian English Voice)
       ↓ (WebRTC Audio)
Browser Speakers (User hears response)
```

• Audio Transport: LiveKit (WebRTC). Handles real-time, low-latency audio streaming between the browser and backend worker.

• VAD (Voice Activity Detection): Silero VAD. Runs locally inside the worker to detect when the user starts and stops speaking.

• STT (Speech-to-Text): Groq Whisper (whisper-large-v3-turbo). Transcribes speech into text in ~100–200ms.

• LLM Reasoning & Tool Calling: Groq (openai/gpt-oss-120b). Fast reasoning and function calling.

• TTS (Text-to-Speech): Cartesia (sonic-turbo). Uses the "Priya" voice profile—a natural, clear Indian English tone suited for a Bengaluru clinic.

• State & Database: SQLite + Python. Manages appointments, availability checks, and audit trails.

• Frontend: React + Vite with @livekit/components-react and Tailwind CSS.

Building the basic happy path is easy; making real-time voice work reliably is where the real learning happened. Here are 4 specific problems I ran into:

When I first ran voice tests, everything would work for 2 or 3 turns, and then suddenly crash with an HTTP 429 Too Many Requests (Rate Limit Exceeded).

Why it happened:

On Groq's free tier, there is an 8,000 Tokens Per Minute (TPM) limit. My initial system prompt combined with the JSON schemas for 5 function tools was taking ~1,900 tokens per single LLM call. If the user spoke 4 times in a minute,

that was 1,900 * 4 = 7,600+ tokens, immediately blowing through the 8,000 token limit.

How I fixed it:

I completely compressed the system prompt and tool docstrings. I removed repetitive instructions, used dense bullet points, and kept the tool parameters minimal while preserving all clinical guardrails. This brought the request size

down from ~1,900 tokens to ~500 tokens (a 73% reduction). Suddenly, I could have 15+ turns a minute without hitting rate limits.

While testing with my laptop mic, I noticed the agent would randomly trigger and start speaking even when I hadn't said anything.

Why it happened:

Default VAD sensitivity was picking up subtle background sounds—fan noise, keyboard typing, and breathing. Each micro-sound triggered LiveKit's turn detector, which immediately dispatched an STT call, an LLM call, and a TTS synthesis

call. This was burning API quota and interrupting the conversation.

How I fixed it:

I tuned the Silero VAD parameters and turn handling in LiveKit:

```
vad_instance = silero.VAD.load(
    min_speech_duration=0.25,   # Ignore clicks and breath sounds under 250ms
    min_silence_duration=0.65,  # Wait for a clean pause before marking end-of-turn
    prefix_padding_duration=0.3,
    activation_threshold=0.6,   # Require clearer vocal energy over ambient noise
)

session = AgentSession(
    turn_handling={
        "endpointing": {"min_delay": 0.6, "max_delay": 3.0},
        "preemptive_generation": {"enabled": False},  # Only generate audio when speech is complete
        "interruption": {"enabled": True, "min_duration": 0.5},
    }
)
```

Disabling preemptive generation and requiring at least 250ms of vocal energy eliminated the false triggers completely.

Initially, I used Groq's built-in TTS. While it worked, it only had US and Arabic voice profiles, and the free-tier rate limits were strict for voice generation.

For a clinic located in HSR Layout, Bengaluru, a North American voice felt out of place. I integrated Cartesia's sonic-turbo model with their Priya voice (an Indian English female voice profile).

The difference was night and day:

• Latency dropped below 100ms.

• The cadence and pronunciation of Indian names sounded authentic.

When I tried deploying the Python backend, I quickly realized you can't just throw a voice agent onto serverless platforms like Vercel or AWS Lambda.

A text chatbot handles a quick HTTP request and terminates in 1 second. A voice agent, on the other hand, is a persistent WebRTC worker daemon. It maintains an active bidirectional audio socket to LiveKit Cloud 24/7.

The Solution:

• React Frontend: Deployed on Vercel with a serverless token endpoint (/api/token) that generates short-lived LiveKit JWT access tokens without exposing LIVEKIT_API_SECRET to the browser.

• Python Agent Worker: Containerized via Docker and deployed to LiveKit Cloud Agent Hosting in the ap-south (Mumbai) region for ultra-low ping.

When deploying the Docker container, I ran into a ModuleNotFoundError: No module named 'agent.prompts'. The container entrypoint was running python agent/agent.py start, which put /app/agent into Python's sys.path instead of the root

/app. I fixed this by adding ENV PYTHONPATH="/app" in the Dockerfile and adding a defensive path resolution in Python.

This project gave me a massive appreciation for what it takes to build reliable real-time AI systems.

I'd love to connect with other engineers and builders working in Voice AI, WebRTC, and LLM tool calling:

• How are you handling ambient noise and interruption handling in your voice agents?

• What TTS providers have you found best for regional accents?

Check out the code and feel free to share your thoughts or suggestions!

🔗 GitHub Repo: [https://github.com/CosmosTechy/maya-ai-voice-receptionist](https://github.com/CosmosTechy/maya-ai-voice-receptionist)

🌐 Live Demo: [https://maya-ai-voice-receptionist.vercel.app/](https://maya-ai-voice-receptionist.vercel.app/)