Building Arogya Seva: How I Built an Ultra-Low Latency Telehealth Voice AI for Bharat in 10 Days

A developer built Arogya Seva, an ultra-low-latency telehealth voice AI for India, in 10 days as part of the #VoiceForBharat challenge. The system uses LiveKit Agents SDK with Deepgram Nova-3 for speech-to-text, Google Gemini 2.0 Flash for intent processing, and Murf Falcon for text-to-speech, supporting multiple Indian languages and scripts. It includes strict clinical guardrails, privacy-first memory with explicit consent, and features like symptom triage, PHC lookup, and human escalation.

Arogya Seva was created to bridge this gap as part of the VoiceForBharat challenge Track: Health Access . It is an empathetic, multilingual, real-time voice assistant designed to interact naturally in Indian English, Hindi Devanagari script , and regional scripts. Why Voice? For millions of non-tech-savvy users or individuals in low-literacy regions, typing in an app or filling out complex forms is a friction point. Speaking directly over a phone call or web interface is the most accessible, natural, and human way to receive guidance. The system is built on LiveKit Agents SDK with a modular pipeline: Speech-to-Text STT : Deepgram Nova-3 transcribes spoken voice in real time. Brain LLM : Google Gemini 2.0 Flash processes intent, applies clinical guardrails, and decides on function tool calls. Text-to-Speech TTS : Murf Falcon livekit-murf plugin, voice model en-IN-Anisha streams ultra-low latency, human-like voice synthesis back to the user. Real-time Transport: LiveKit WebRTC web frontend and SIP Telephony outbound/inbound phone calls . Memory & State: SQLite agent memory.db for privacy-first caller persistence and escalation management. Mermaid diagram 🛡️ Feature 2: Strict Guardrails & Native Script Enforcement Health AI requires absolute safety. Arogya Seva follows strict operational boundaries: Red-Flag Clinical Emergency Protocol: Immediately flags chest pain, dyspnea, heavy bleeding, or acute trauma, urging callers to dial emergency 108. Native Script Enforcement: To ensure proper acoustic synthesis and avoid awkward transliteration, responses in Hindi are strictly produced in native Devanagari script e.g., नमस्ते, आप कैसे हैं? , avoiding romanized "Hinglish". 💻 Feature 3: Dynamic Frontend State & Audio Visualizers Built with Next.js and LiveKit Agents UI, the frontend displays real-time agent states: Listening Visualized with dynamic frequency waveforms Thinking Tool execution state Speaking Fluid audio spectrum representation 🧠 Feature 4: Privacy-First Memory with Explicit Consent Returning callers don't need to re-explain their location or age band. However, privacy is paramount: The agent explicitly asks: "May I save your name and basic health details so I can remember you next time?" Facts are stored only if explicit consent is given. Users can say "Forget me" at any time to wipe their records via forget caller. 🛠️ Feature 5: Real-Domain Health Tools & Tool Chaining classify symptom triage: Categorizes symptoms into Self-Care / Low, Moderate / Consult Nurse, or High Urgent / Red-Flag. lookup nearest phc: Searches Primary Health Centres based on district. Tool Chaining: Automatically reuses district information saved in caller memory without re-asking the user. Graceful Failure: If the registry API is unreachable, the agent announces the offline status calmly and provides emergency helpline 104/108 numbers. 📞 Feature 6: Outbound Telephony & Mandatory Opt-Out For automated health reminders and follow-up calls: Two-Sentence Mandatory Opening: State WHO is calling, WHY, and HOW to opt out in the first two sentences. Instant Opt-Out: Saying "stop calling me" or pressing 9 immediately executes opt out caller in SQLite and terminates the call. 🆘 Feature 7: Human Escalation & Reference IDs When situations exceed AI scope: Agent detects clinical doctor requests or red-flag symptoms. Agent requests explicit permission to create an escalation ticket. Upon agreement, create escalation stores a sanitized summary no passwords/PINs/Aadhaar and returns a unique reference ID e.g., ESC-8492 . 📊 Feature 8: Call Analytics & Outcome Tracking Every call session logs structured metrics into SQLite, including call duration, triage classifications, escalation status, and resolution codes triage completed, phc found, escalated, handed off . 🔀 Feature 9: Multi-Agent Specialist Handoff When callers request to schedule, modify, or cancel OPD appointments, the main agent invokes transfer to clinic specialist: python @function tool async def transfer to clinic specialist self, context: RunContext, reason: str - str: specialist = ClinicAppointmentSpecialist context.session.update agent specialist return "Handed off conversation to Clinic and Appointment Specialist." The session dynamically updates to ClinicAppointmentSpecialist, seamlessly swapping persona and toolsets without dropping the audio call Step 1: Prerequisites Python 3.10+ & uv package manager Node.js 18+ & pnpm LiveKit Cloud account URL, API Key, API Secret Murf AI API Key for Falcon TTS Deepgram API Key for STT Google Gemini API Key for LLM Step 2: Clone & Configure Backend bash git clone https://github.com/viral-1998/VoiceOfBharat.git https://github.com/viral-1998/VoiceOfBharat.git cd VoiceOfBharat/backend cp .env.example .env.local Add your API keys to backend/.env.local: env LIVEKIT URL=wss://your-livekit-project.livekit.cloud LIVEKIT API KEY=your key LIVEKIT API SECRET=your secret MURF API KEY=your murf key DEEPGRAM API KEY=your deepgram key GOOGLE API KEY=your google key Step 3: Run Backend Agent bash uv sync uv run python src/agent.py download-files First time model download uv run python src/agent.py dev Start live dev server Step 4: Run Frontend UI bash cd ../frontend pnpm install pnpm dev Open http://localhost:3000 http://localhost:3000 in your browser, click Connect, and start speaking to your agent python @function tool async def transfer to clinic specialist self, context: RunContext, reason: str = "User requested appointment booking", - str: """Transfer caller to Clinic & Appointment Specialist agent.""" specialist = ClinicAppointmentSpecialist context.session.update agent specialist call id = getattr getattr context, "session", None , "call id", "" if call id: db.mark call success call id, outcome summary=f"Handed off: {reason}" return "Handed off conversation to Clinic and Appointment Specialist." Multi-lingual Voice Cloning: Adding localized voice accents across 10+ Indian regional languages using Murf Falcon's voice library. WhatsApp Telemetry Notifications: Sending automated SMS/WhatsApp appointment receipts following human escalations. EHR Integration: Connecting triage outcomes directly with ABDM Ayushman Bharat Digital Mission health IDs.