# Building Arogya Seva: How I Built an Ultra-Low Latency Telehealth Voice AI for Bharat in 10 Days > Source: > Published: 2026-08-15 11:20:47+00:00 Arogya Seva was created to bridge this gap as part of the #VoiceForBharat challenge (Track: Health Access). It is an empathetic, multilingual, real-time voice assistant designed to interact naturally in Indian English, Hindi (Devanagari script), and regional scripts. Why Voice? For millions of non-tech-savvy users or individuals in low-literacy regions, typing in an app or filling out complex forms is a friction point. Speaking directly over a phone call or web interface is the most accessible, natural, and human way to receive guidance. The system is built on LiveKit Agents SDK with a modular pipeline: Speech-to-Text (STT): Deepgram Nova-3 transcribes spoken voice in real time. Brain (LLM): Google Gemini 2.0 Flash processes intent, applies clinical guardrails, and decides on function tool calls. Text-to-Speech (TTS): Murf Falcon (livekit-murf plugin, voice model en-IN-Anisha) streams ultra-low latency, human-like voice synthesis back to the user. Real-time Transport: LiveKit WebRTC (web frontend) and SIP Telephony (outbound/inbound phone calls). Memory & State: SQLite (agent_memory.db) for privacy-first caller persistence and escalation management. Mermaid diagram 🛡️ Feature 2: Strict Guardrails & Native Script Enforcement Health AI requires absolute safety. Arogya Seva follows strict operational boundaries: Red-Flag Clinical Emergency Protocol: Immediately flags chest pain, dyspnea, heavy bleeding, or acute trauma, urging callers to dial emergency 108. Native Script Enforcement: To ensure proper acoustic synthesis and avoid awkward transliteration, responses in Hindi are strictly produced in native Devanagari script (e.g., नमस्ते, आप कैसे हैं?), avoiding romanized "Hinglish". 💻 Feature 3: Dynamic Frontend State & Audio Visualizers Built with Next.js and LiveKit Agents UI, the frontend displays real-time agent states: Listening (Visualized with dynamic frequency waveforms) Thinking (Tool execution state) Speaking (Fluid audio spectrum representation) 🧠 Feature 4: Privacy-First Memory with Explicit Consent Returning callers don't need to re-explain their location or age band. However, privacy is paramount: The agent explicitly asks: "May I save your name and basic health details so I can remember you next time?" Facts are stored only if explicit consent is given. Users can say "Forget me" at any time to wipe their records via forget_caller. 🛠️ Feature 5: Real-Domain Health Tools & Tool Chaining classify_symptom_triage: Categorizes symptoms into Self-Care / Low, Moderate / Consult Nurse, or High Urgent / Red-Flag. lookup_nearest_phc: Searches Primary Health Centres based on district. Tool Chaining: Automatically reuses district information saved in caller memory without re-asking the user. Graceful Failure: If the registry API is unreachable, the agent announces the offline status calmly and provides emergency helpline 104/108 numbers. 📞 Feature 6: Outbound Telephony & Mandatory Opt-Out For automated health reminders and follow-up calls: Two-Sentence Mandatory Opening: State WHO is calling, WHY, and HOW to opt out in the first two sentences. Instant Opt-Out: Saying "stop calling me" or pressing 9 immediately executes opt_out_caller in SQLite and terminates the call. 🆘 Feature 7: Human Escalation & Reference IDs When situations exceed AI scope: Agent detects clinical doctor requests or red-flag symptoms. Agent requests explicit permission to create an escalation ticket. Upon agreement, create_escalation stores a sanitized summary (no passwords/PINs/Aadhaar) and returns a unique reference ID (e.g., ESC-8492). 📊 Feature 8: Call Analytics & Outcome Tracking Every call session logs structured metrics into SQLite, including call duration, triage classifications, escalation status, and resolution codes (triage_completed, phc_found, escalated, handed_off). 🔀 Feature 9: Multi-Agent Specialist Handoff When callers request to schedule, modify, or cancel OPD appointments, the main agent invokes transfer_to_clinic_specialist: python @function_tool async def transfer_to_clinic_specialist(self, context: RunContext, reason: str) -> str: specialist = ClinicAppointmentSpecialist() context.session.update_agent(specialist) return "Handed off conversation to Clinic and Appointment Specialist." The session dynamically updates to ClinicAppointmentSpecialist, seamlessly swapping persona and toolsets without dropping the audio call! Step 1: Prerequisites Python 3.10+ & uv package manager Node.js 18+ & pnpm LiveKit Cloud account (URL, API Key, API Secret) Murf AI API Key (for Falcon TTS) Deepgram API Key (for STT) Google Gemini API Key (for LLM) Step 2: Clone & Configure Backend bash git clone [https://github.com/viral-1998/VoiceOfBharat.git](https://github.com/viral-1998/VoiceOfBharat.git) cd VoiceOfBharat/backend cp .env.example .env.local Add your API keys to backend/.env.local: env LIVEKIT_URL=wss://your-livekit-project.livekit.cloud LIVEKIT_API_KEY=your_key LIVEKIT_API_SECRET=your_secret MURF_API_KEY=your_murf_key DEEPGRAM_API_KEY=your_deepgram_key GOOGLE_API_KEY=your_google_key Step 3: Run Backend Agent bash uv sync uv run python src/agent.py download-files # First time model download uv run python src/agent.py dev # Start live dev server Step 4: Run Frontend UI bash cd ../frontend pnpm install pnpm dev Open [http://localhost:3000](http://localhost:3000) in your browser, click Connect, and start speaking to your agent! python @function_tool async def transfer_to_clinic_specialist( self, context: RunContext, reason: str = "User requested appointment booking", ) -> str: """Transfer caller to Clinic & Appointment Specialist agent.""" specialist = ClinicAppointmentSpecialist() context.session.update_agent(specialist) ``` call_id = getattr(getattr(context, "session", None), "call_id", "") if call_id: db.mark_call_success(call_id, outcome_summary=f"Handed off: {reason}") return "Handed off conversation to Clinic and Appointment Specialist." ``` Multi-lingual Voice Cloning: Adding localized voice accents across 10+ Indian regional languages using Murf Falcon's voice library. WhatsApp Telemetry Notifications: Sending automated SMS/WhatsApp appointment receipts following human escalations. EHR Integration: Connecting triage outcomes directly with ABDM (Ayushman Bharat Digital Mission) health IDs.