Arogya Seva was created to bridge this gap as part of the #VoiceForBharat challenge (Track: Health Access). It is an empathetic, multilingual, real-time voice assistant designed to interact naturally in Indian English, Hindi (Devanagari script), and regional scripts.
Why Voice? For millions of non-tech-savvy users or individuals in low-literacy regions, typing in an app or filling out complex forms is a friction point. Speaking directly over a phone call or web interface is the most accessible, natural, and human way to receive guidance.
The system is built on LiveKit Agents SDK with a modular pipeline:
Speech-to-Text (STT): Deepgram Nova-3 transcribes spoken voice in real time.
Brain (LLM): Google Gemini 2.0 Flash processes intent, applies clinical guardrails, and decides on function tool calls.
Text-to-Speech (TTS): Murf Falcon (livekit-murf plugin, voice model en-IN-Anisha) streams ultra-low latency, human-like voice synthesis back to the user.
Real-time Transport: LiveKit WebRTC (web frontend) and SIP Telephony (outbound/inbound phone calls).
Memory & State: SQLite (agent_memory.db) for privacy-first caller persistence and escalation management.
Mermaid diagram
🛡️ Feature 2: Strict Guardrails & Native Script Enforcement
Health AI requires absolute safety. Arogya Seva follows strict operational boundaries:
Red-Flag Clinical Emergency Protocol: Immediately flags chest pain, dyspnea, heavy bleeding, or acute trauma, urging callers to dial emergency 108.
Native Script Enforcement: To ensure proper acoustic synthesis and avoid awkward transliteration, responses in Hindi are strictly produced in native Devanagari script (e.g., नमस्ते, आप कैसे हैं?), avoiding romanized "Hinglish".
💻 Feature 3: Dynamic Frontend State & Audio Visualizers
Built with Next.js and LiveKit Agents UI, the frontend displays real-time agent states:
Listening (Visualized with dynamic frequency waveforms)
Thinking (Tool execution state)
Speaking (Fluid audio spectrum representation)
🧠 Feature 4: Privacy-First Memory with Explicit Consent
Returning callers don't need to re-explain their location or age band. However, privacy is paramount:
The agent explicitly asks: "May I save your name and basic health details so I can remember you next time?"
Facts are stored only if explicit consent is given.
Users can say "Forget me" at any time to wipe their records via forget_caller.
🛠️ Feature 5: Real-Domain Health Tools & Tool Chaining
classify_symptom_triage: Categorizes symptoms into Self-Care / Low, Moderate / Consult Nurse, or High Urgent / Red-Flag.
lookup_nearest_phc: Searches Primary Health Centres based on district.
Tool Chaining: Automatically reuses district information saved in caller memory without re-asking the user.
Graceful Failure: If the registry API is unreachable, the agent announces the offline status calmly and provides emergency helpline 104/108 numbers.
📞 Feature 6: Outbound Telephony & Mandatory Opt-Out
For automated health reminders and follow-up calls:
Two-Sentence Mandatory Opening: State WHO is calling, WHY, and HOW to opt out in the first two sentences.
Instant Opt-Out: Saying "stop calling me" or pressing 9 immediately executes opt_out_caller in SQLite and terminates the call.
🆘 Feature 7: Human Escalation & Reference IDs
When situations exceed AI scope:
Agent detects clinical doctor requests or red-flag symptoms.
Agent requests explicit permission to create an escalation ticket.
Upon agreement, create_escalation stores a sanitized summary (no passwords/PINs/Aadhaar) and returns a unique reference ID (e.g., ESC-8492).
📊 Feature 8: Call Analytics & Outcome Tracking
Every call session logs structured metrics into SQLite, including call duration, triage classifications, escalation status, and resolution codes (triage_completed, phc_found, escalated, handed_off).
🔀 Feature 9: Multi-Agent Specialist Handoff
When callers request to schedule, modify, or cancel OPD appointments, the main agent invokes transfer_to_clinic_specialist:
python
@function_tool
async def transfer_to_clinic_specialist(self, context: RunContext, reason: str) -> str:
specialist = ClinicAppointmentSpecialist()
context.session.update_agent(specialist)
return "Handed off conversation to Clinic and Appointment Specialist."
The session dynamically updates to ClinicAppointmentSpecialist, seamlessly swapping persona and toolsets without dropping the audio call!
Step 1: Prerequisites
Python 3.10+ & uv package manager
Node.js 18+ & pnpm
LiveKit Cloud account (URL, API Key, API Secret)
Murf AI API Key (for Falcon TTS)
Deepgram API Key (for STT)
Google Gemini API Key (for LLM)
Step 2: Clone & Configure Backend
bash
git clone https://github.com/viral-1998/VoiceOfBharat.git
cd VoiceOfBharat/backend
cp .env.example .env.local
Add your API keys to backend/.env.local:
env
LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_key
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_google_key
Step 3: Run Backend Agent
bash
uv sync
uv run python src/agent.py download-files # First time model download
uv run python src/agent.py dev # Start live dev server
Step 4: Run Frontend UI
bash
cd ../frontend
pnpm install
pnpm dev
Open http://localhost:3000 in your browser, click Connect, and start speaking to your agent!
python
@function_tool
async def transfer_to_clinic_specialist(
self,
context: RunContext,
reason: str = "User requested appointment booking",
) -> str:
"""Transfer caller to Clinic & Appointment Specialist agent."""
specialist = ClinicAppointmentSpecialist()
context.session.update_agent(specialist)
call_id = getattr(getattr(context, "session", None), "call_id", "")
if call_id:
db.mark_call_success(call_id, outcome_summary=f"Handed off: {reason}")
return "Handed off conversation to Clinic and Appointment Specialist."
Multi-lingual Voice Cloning: Adding localized voice accents across 10+ Indian regional languages using Murf Falcon's voice library.
WhatsApp Telemetry Notifications: Sending automated SMS/WhatsApp appointment receipts following human escalations.
EHR Integration: Connecting triage outcomes directly with ABDM (Ayushman Bharat Digital Mission) health IDs.