Building Kisan Mitra: How I Built an Ultra-Fast Voice AI for Indian Farmers in 10 Days

A developer built Kisan Mitra, an ultra-fast multilingual voice AI assistant for Indian farmers, in 10 days. The system uses LiveKit, Deepgram Nova-3, Gemini 2.5 Flash, and Murf Falcon to provide real-time mandi prices, weather updates, and proactive price alerts via phone calls, with features like caller memory and human escalation.

From zero to a full-stack, multilingual agricultural voice agent with caller memory, real-time mandi tools, outbound price alert calls, human escalation, and specialist agent handoffs — powered by Murf Falcon & LiveKit. 🌟 The Problem & The Mission In rural India, millions of farmers make critical livelihood decisions every day: When should I harvest? Will it rain before I spray pesticides? Which nearby mandi market is offering the best price for my cotton crop? While agricultural data exists across various portals, accessing it through complex web interfaces or text-heavy apps is challenging for farmers out in the field. Voice is the natural, frictionless interface for Bharat. A farmer standing in an orchard or driving a tractor doesn't want to type queries into a search bar; they want to speak naturally in their native language or conversational Hinglish and get instant, reliable answers. For the 10 Days of Voice Agents VoiceForBharat Edition , I chose the Farm & Field track and built Kisan Mitra किसान मित्र — an empathetic, real-time AI voice assistant tailored specifically for Indian agriculture. 🏗️ Architecture & Core Components A production-grade voice agent is fundamentally different from a text chatbot. Latency is the single biggest factor in conversational realism: if the agent takes more than 1–1.5 seconds to reply, the human conversation breaks down. mermaid flowchart LR A 🎙️ Farmer Speaks -- |Audio Stream| B Deepgram Nova-3 STT B -- |Transcribed Text| C Gemini 2.5 Flash LLM C -- |Streamed Tokens| D Murf Falcon TTS D -- |Real-time Audio| E LiveKit WebRTC E -- |Ultra-low Latency Audio| F 🔊 Farmer Hears Answer C <-- |Tools & Memory| G SQLite & External APIs The 4 Pillars of the Pipeline: Real-time Transport LiveKit : Manages ultra-low-latency, bidirectional audio WebRTC streaming and turn detection. Speech-to-Text Deepgram Nova-3 : Accurately transcribes spoken Indian English and accented Hindi. LLM Brain Google Gemini 2.5 Flash : Handles intent detection, domain reasoning, guardrails, and tool calling. Fast Text-to-Speech Murf Falcon : The game changer. With sub-100ms time-to-first-audio, Murf Falconstreams natural, warm Indian voices Anisha / hi-IN without robotic pauses. 🚀 Key Features Built Across the 10 Days Conversational, concise spoken responses no markdown syntax or raw JSON read aloud . Absolute refusal to fabricate market rates or weather data. lookup mandi prices crop, district : Queries market prices across key Indian APMCs e.g., Yavatmal, Nagpur, Lasalgaon and explicitly cites timestamps e.g., "As of today's Agmarknet live update..." . get district weather district : Leverages Open-Meteo live satellite feeds to deliver temperature, rain probability, and actionable agronomic advice e.g., "Rain probability is 65% today; postpone chemical spraying" . Graceful Failure & Out-Loud Transparency External APIs fail in the real world. Rather than hallucinating rates or hanging silently, Kisan Mitra catches timeouts and announces the service outage out loud to the caller: Caller Memory & Privacy-First Persistence Kisan Mitra remembers returning farmers e.g., their land size, crops grown, district across sessions via SQLite. Crucial guardrail: The agent never saves data without first asking: "May I save these details so I can remember you for our next call?" Outbound Telephony & Proactive Price Alerts When mandi rates cross a farmer's predefined threshold e.g., Cotton crossing ₹7,000/quintal in Yavatmal , Kisan Mitra autonomously places a phone call via Twilio & TwiML: Compliance in the first 2 sentences: Explains who is calling, why, and how to opt out Press 9 to unsubscribe, 1 for details . Outcome tracking: Automatically handles busy lines, no-answers retry in 2h , and short hang-ups. Human Escalation Krishi Vigyan Kendra Officer Support For emergencies severe pest attacks like Pink Bollworm or crop blight , the agent prompts the farmer for consent and logs a structured escalation ticket with sanitized PII, assigning a reference number like ESC-48291 for agricultural officer callbacks. Specialist Agent Handoffs & Analytics Dashboard Handoffs: Complex agronomic pathology queries are handed off seamlessly to a dedicated CropSpecialist sub-agent. Analytics: Complete call outcome tracking completed, success, failed, reason stored in SQLite and visualizable via an admin dashboard. 🛠️ The Hardest Challenges & Lessons Learned Eliminating Conversational Latency The Problem: Combining STT + LLM reasoning + TTS synthesis often introduces awkward 2–3 second silences. The Solution: Streaming tokens incrementally from Gemini into Murf Falcon using SentenceTokenizer min sentence len=2 and preemptive generation=True. Audio synthesis begins before the LLM finishes generating the full paragraph. Handling Code-Mixed Hindi & Indian English Hinglish The Problem: Standard VAD Voice Activity Detection models often cut off speakers mid-sentence when Indian language filler words "haanji", "achha", "matlab" were used. The Solution: Integrated LiveKit's MultilingualModel turn detector paired with Silero VAD to maintain natural listening rhythms. Out-Loud Tool Failures vs. Silent Errors The Problem: When an external weather API timed out, the LLM initially tended to guess yesterday's temperature. The Solution: Enforced strict system instructions where tools return explicit FAILURE strings that instruct the LLM: "State out loud that the service is unreachable right now. Do not guess." GITHUB :- https://github.com/codebynikhil08/murf-livekit-starter https://github.com/codebynikhil08/murf-livekit-starter