{"slug": "ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response", "title": "Ashraya AI: My 10-Day Journey Building a Voice Agent for Disaster Response", "summary": "An engineer built Ashraya AI, a voice-based disaster-response assistant, during the 10 Days of Voice Agents challenge by Murf AI. The system uses a multi-agent architecture with specialist handoffs, memory, and safety boundaries to guide users during emergencies while avoiding hallucinated information.", "body_md": "From a simple voice assistant to a multi-agent disaster-response system\n\nDisasters don't wait for people to open an app, read instructions, or type a message.\n\nIn an emergency, people need information quickly—and sometimes they simply need to talk.\n\nThat was the idea behind Ashraya AI, the disaster-response voice agent I built during the 10 Days of Voice Agents — VoiceForBharat Edition challenge by Murf AI.\n\nOver the past 10 days, I went from building a basic conversational voice agent to developing a system with memory, tools, outbound calling, call analytics, success tracking, human escalation, and specialist-agent handoffs.\n\nThe goal was not to build another chatbot that can answer everything.\n\nThe goal was to build a voice-first system that can guide people during disaster situations while being explicit about what it does not know.\n\n**1. The Problem**\n\nDuring disasters such as floods, earthquakes, cyclones, or other emergencies, information can become difficult to access.\n\nA person may need answers to questions like:\n\nWhat should I do right now?\n\nWhat should I take if I need to leave?\n\nWhere can I get help?\n\nHow should I prepare an elderly family member for evacuation?\n\nCan someone help me if I cannot handle the situation myself?\n\nA traditional text interface isn't always ideal in these situations.\n\nSomeone may be frightened, busy helping family members, unable to type comfortably, or simply need to communicate naturally.\n\nThat's where a voice interface can be useful.\n\nAshraya AI is designed as a conversational disaster-response assistant that users can interact with through voice.\n\nThe project focuses on providing practical guidance while maintaining strict safety boundaries.\n\n**2. Why Voice?**\n\nVoice changes the interaction model.\n\nInstead of navigating menus or typing questions, a user can simply say:\n\n\"There is flooding near my house. What should I do?\"\n\nor:\n\n\"I may need to go to a shelter. What should I take?\"\n\nThe agent can respond conversationally and ask relevant follow-up questions.\n\nFor a disaster-response use case, this matters because the user's attention may already be occupied by the emergency itself.\n\nBut voice introduces another challenge:\n\nAn AI that sounds confident can easily sound authoritative—even when it is wrong.\n\nThat made safety and hallucination prevention one of the most important parts of this project.\n\n**3. What is Ashraya AI?**\n\nAshraya AI is a voice-based disaster-response assistant built using:\n\nMurf Falcon — text-to-speech\n\nLiveKit Agents — real-time voice-agent infrastructure\n\nDeepgram — speech-to-text\n\nGemini — language model\n\nPython — application logic and agent orchestration\n\nThe architecture evolved throughout the challenge.\n\nInstead of treating Ashraya as one giant agent that handles every possible question, I eventually introduced a specialist-agent architecture.\n\nThe main agent handles the overall disaster-response conversation.\n\nWhen the conversation becomes specifically shelter-related, it can hand the conversation to a Shelter Information Specialist.\n\nThis creates a much clearer separation of responsibilities.\n\n**4. System Architecture**\n\nAt a high level, the voice interaction works like this:\n\n```\n                User\n                  │\n                  ▼\n            Microphone\n                  │\n                  ▼\n          Speech-to-Text\n            (Deepgram)\n                  │\n                  ▼\n          Main AI Agent\n            (Gemini)\n                  │\n      ┌───────────┴───────────┐\n      │                       │\n      ▼                       ▼\nGeneral disaster       Shelter-related\n   assistance             request\n                              │\n                              ▼\n                 Shelter Information\n                     Specialist\n                              │\n                              ▼\n                     Shelter Guidance\n                  (without fabricated\n                   live information)\n                              │\n                              ▼\n                     Murf Falcon TTS\n                              │\n                              ▼\n                           User\n```\n\nThe important design decision here is that the specialist is not simply another prompt inside the main agent.\n\nIt has its own instructions, scope, limitations, and success tracking.\n\n**5. Building the Voice Agent**\n\nA real-time voice agent fundamentally needs several components working together:\n\nSpeech-to-Text\n\nThe user's speech needs to be converted into text so the language model can understand it.\n\nI used Deepgram for speech recognition.\n\nLanguage Model\n\nThe language model handles reasoning, conversation, tool selection, and responses.\n\nI used Gemini for this layer.\n\nText-to-Speech\n\nThe generated response needs to be converted back into natural speech.\n\nFor this project, I used Murf Falcon.\n\nReal-Time Transport\n\nThe system needs a real-time communication layer to move audio between the user and the agent.\n\nI used LiveKit Agents.\n\nTogether, these components form the basic voice-agent pipeline:\n\nUser Speech\n\n↓\n\nSpeech-to-Text\n\n↓\n\nLLM / Agent Logic\n\n↓\n\nText-to-Speech\n\n↓\n\nSpoken Response\n\n**6. Giving the Agent a Clear Personality and Safety Boundary**\n\nOne of the first lessons I learned was that an agent's prompt is not just about telling it what to say.\n\nIt also needs to define:\n\nWhat it is allowed to do\n\nWhat it is not allowed to do\n\nWhen it should ask questions\n\nWhen it should use a tool\n\nWhen it should escalate\n\nWhat information it must never invent\n\nFor a disaster-response system, this becomes especially important.\n\nFor example, an unsafe agent might respond to:\n\n\"Which government shelter is open near me?\"\n\nwith a fabricated answer such as:\n\n\"The nearest shelter is XYZ School.\"\n\nThat would be unacceptable.\n\nAshraya AI is explicitly instructed not to invent live shelter locations, availability, capacity, evacuation routes, or government orders when it doesn't have verified real-time data.\n\nInstead, it should clearly state the limitation and direct the caller toward appropriate official local authorities.\n\nThat distinction became one of the most important safety principles in the project:\n\nNot knowing something is better than confidently making something up.\n\n**7. Memory for Returning Users**\n\nAnother part of the project was giving Ashraya AI memory.\n\nThe idea was to avoid treating every conversation as if it were happening with a completely new user.\n\nMemory can allow the system to retain useful conversational context and provide more consistent interactions when appropriate.\n\nHowever, memory in a disaster-response system also needs to be handled carefully.\n\nThe agent should use relevant context without making assumptions about the user's current emergency.\n\nCurrent information from the caller should always take priority over stale conversational context.\n\n**8. Tools and Actions**\n\nThe agent is not limited to generating text.\n\nTools allow it to perform specific actions.\n\nThis is important because an AI agent should not rely on the language model to simulate actions that should actually happen in software.\n\nFor example, instead of simply saying:\n\n\"I'll connect you to a specialist.\"\n\nthe system can invoke a handoff tool that actually transfers the conversation.\n\nThis became particularly important during Day 9.\n\n**9. Multi-Agent Handoff: The Shelter Specialist**\n\nOne of the biggest architectural changes was introducing a dedicated Shelter Information Specialist.\n\nThe main Ashraya AI agent is broader.\n\nThe shelter specialist is intentionally narrow.\n\nIt focuses on questions such as:\n\nWhat should I take to a shelter?\n\nHow should I prepare before moving?\n\nWhat documents should I carry?\n\nWhat medicines or essential supplies should I take?\n\nWhat should I consider when travelling with children or older adults?\n\nWhat information should I obtain from local authorities?\n\nBut it is not a general disaster-response agent.\n\nIf someone asks the specialist:\n\n\"What should I do during an earthquake?\"\n\nor:\n\n\"How do I perform CPR?\"\n\nthe specialist should not suddenly become a medical or earthquake-response expert.\n\nIt should say that the question is outside its role and redirect the user appropriately.\n\nThis is a subtle but important difference between a multi-agent system and a single agent with a very large prompt.\n\nThe goal isn't:\n\n\"Make every agent know everything.\"\n\nThe goal is:\n\n\"Give each agent a clear responsibility and let the system decide who should handle the conversation.\"\n\n**10. Preventing Hallucinated Shelter Information**\n\nThis was one of the most important safety challenges in the project.\n\nSuppose someone asks:\n\n\"Which government shelter is open near me right now?\"\n\nIf the system doesn't have a verified live shelter database, it cannot honestly answer that question.\n\nSo the Shelter Information Specialist explicitly states that it does not have verified real-time shelter-location or opening-status information.\n\nIt can instead explain how the caller can obtain verified information from appropriate local authorities.\n\nThis is an important design principle for real-world AI systems:\n\nThe absence of information should produce a limitation—not a hallucination.\n\nA disaster-response assistant should be conservative when dealing with information that can directly affect someone's physical safety.\n\n**11. Outbound Calling**\n\nAshraya AI was also designed to support outbound voice interactions.\n\nThis changes the experience from:\n\n\"A user opens an AI application.\"\n\nto:\n\n\"The AI can initiate a voice conversation.\"\n\nFor a disaster-response scenario, outbound communication could eventually be useful for alerting people, checking whether they need assistance, or providing information.\n\nHowever, outbound calling also introduces additional considerations around consent, privacy, call reliability, and responsible use.\n\n**12. Human Escalation**\n\nAnother important capability was allowing the system to recognize when AI should not be the final layer of support.\n\nThere are situations where a human operator or emergency service may be more appropriate.\n\nThe principle is simple:\n\nAI should assist humans, not pretend to replace emergency infrastructure.\n\nThe agent should never claim that emergency personnel are already responding unless the system actually knows that.\n\n**13. Call Analytics and Success Tracking**\n\nAnother lesson from the project was that a conversation isn't necessarily successful simply because the AI produced a response.\n\nI added call outcome tracking so conversations can be evaluated based on whether the intended objective was actually completed.\n\nFor example, a shelter-related interaction should only be marked successful after meaningful shelter guidance has actually been provided.\n\nA transfer alone should not automatically count as success.\n\nThis distinction is important for evaluating voice agents in production.\n\nYou need to measure outcomes, not just conversations.\n\n**14. The Hardest Part: Making the Agent Stay in Its Lane**\n\nOne of the biggest challenges I encountered was getting the specialist-agent architecture to behave correctly.\n\nAt one point, the Shelter Information Specialist was still answering questions about earthquakes and CPR.\n\nThat was a clear architectural failure.\n\nThe specialist had been created, but its boundaries were not being enforced strongly enough.\n\nI tightened its instructions so that it explicitly refuses questions outside its shelter-information scope.\n\nThis taught me an important lesson:\n\nCreating a specialist agent isn't enough. You have to define its boundaries as carefully as its capabilities.\n\n**15. Debugging the Handoff**\n\nAnother issue I encountered was a technical handoff failure.\n\nThe main agent successfully called the shelter-transfer tool, but the specialist failed to initialize because of an incorrect prompt variable reference.\n\nThe result was a NameError during the transfer.\n\nThe conversation looked correct from the user's perspective until the actual handoff happened—but internally, the specialist could not be created.\n\nThe problem wasn't with the AI reasoning.\n\nIt was a straightforward software integration issue.\n\nThat distinction matters when debugging agentic systems.\n\nSometimes the model isn't the problem.\n\nYour orchestration code is.\n\nAfter correcting the specialist initialization and prompt reference, the handoff could work as intended.\n\n**16. Another Important Lesson: \"Transfer\" Must Actually Transfer**\n\nOne of the most interesting things I learned was that saying:\n\n\"I'm connecting you to the Shelter Information Specialist.\"\n\ndoesn't mean the transfer actually happened.\n\nThe application needs to:\n\nDetect the appropriate intent.\n\nInvoke the transfer tool.\n\nCreate or activate the specialist agent.\n\nPass the relevant conversation context.\n\nReturn control to the specialist.\n\nEnsure the specialist follows its own instructions.\n\nIf any of these steps fail, the main agent may continue speaking.\n\nThat creates a confusing experience where the user asks:\n\n\"Who am I speaking to?\"\n\nand the system gives inconsistent answers.\n\nThis was a useful reminder that agent handoffs are software orchestration problems as much as they are prompting problems.\n\n**17. How to Run the Project**\n\nThe project is built with Python and LiveKit.\n\nAfter cloning the repository, install the required dependencies and configure the required environment variables.\n\nThe API keys should be stored in an environment file such as:\n\n```\n.env\n```\n\nand should never be committed to GitHub.\n\nA typical setup looks like:\n\n```\ngit clone https://github.com/AsmiRaut23/murf-livekit-starter\ncd murf-livekit-starter\npython -m venv .venv\n```\n\nActivate the environment and install the project's dependencies:\n\n```\npip install -r requirements.txt\n```\n\nThen configure the required credentials in your environment variables.\n\nFor example:\n\n```\nLIVEKIT_URL=...\nLIVEKIT_API_KEY=...\nLIVEKIT_API_SECRET=...\n\nMURF_API_KEY=...\nDEEPGRAM_API_KEY=...\nGEMINI_API_KEY=...\n```\n\nThe exact variables depend on the project's implementation.\n\nOnce the environment is configured, start the LiveKit agent using the project's entry point.\n\nDo not publish real API keys, phone numbers, caller information, or other private credentials in the repository or blog.\n\n**18. Testing the Agent**\n\nThe easiest way to test a voice agent is through real conversations rather than only checking whether the code runs.\n\nFor Ashraya AI, I tested scenarios such as:\n\nGeneral disaster question\n\n\"There is flooding near my house. What should I do?\"\n\nShelter question\n\n\"I may need to go to a shelter. What should I take?\"\n\nLive-information limitation\n\n\"Which government shelter is open near me right now?\"\n\nSpecialist boundary\n\nAfter transferring to the Shelter Information Specialist:\n\n\"What should I take to the shelter?\"\n\nand then testing an unrelated question such as:\n\n\"How do I perform CPR?\"\n\nThe specialist should not start behaving like a general medical assistant.\n\nTesting these boundary cases was just as important as testing successful conversations.\n\n**19. What I Would Improve Next**\n\nAshraya AI is still a prototype, and there are several areas I would improve before treating it as a production disaster-response system.\n\nVerified real-time shelter data\n\nThe biggest improvement would be connecting the shelter specialist to an authoritative, continuously updated government or disaster-management data source.\n\nThat would allow the system to answer location-specific shelter questions based on verified information rather than simply explaining that live data is unavailable.\n\nBetter emergency escalation\n\nA production version would need carefully designed integration with emergency services and human operators.\n\nMore robust multilingual support\n\nIndia has enormous linguistic diversity.\n\nExpanding reliable multilingual and code-mixed conversations would make the system more accessible.\n\nStronger evaluation\n\nI would also build a larger evaluation suite covering:\n\nHallucination resistance\n\nSpecialist boundaries\n\nEmergency scenarios\n\nTool failures\n\nHandoff failures\n\nCall completion\n\nLatency\n\nFalse transfers\n\nSuccessful outcomes\n\n**20. What I Learned in 10 Days**\n\nThe biggest lesson from this challenge wasn't how to connect speech-to-text to an LLM and then to text-to-speech.\n\nIt was learning that a useful voice agent is much more than a model that can talk.\n\nA reliable system needs:\n\nConversation + tools + memory + guardrails + orchestration + observability + failure handling.\n\nAnd sometimes the most important behavior of an AI agent is knowing when not to answer.\n\nBuilding Ashraya AI also changed how I think about multi-agent systems.\n\nI initially thought the challenge was to make one agent increasingly capable.\n\nBy the end, I realized that a better approach is often to build a system of focused agents with clear responsibilities.\n\nFinal Thoughts\n\nOver these 10 days, Ashraya AI evolved from a voice assistant into a more structured disaster-response system.\n\nIt can:\n\nHave natural voice conversations\n\nUse an Indian voice powered by Murf Falcon\n\nFollow explicit safety guardrails\n\nMaintain conversational context\n\nUse tools\n\nSupport outbound calling\n\nTrack call outcomes\n\nEscalate when necessary\n\nHand shelter-related conversations to a specialist agent\n\nRefuse to fabricate information it cannot verify\n\nThe project is still a work in progress.\n\nAnd that's probably the most valuable part of the experience.\n\nBuilding a voice agent isn't about making AI sound impressive.\n\nIt's about making the system useful, predictable, and honest about its limitations.\n\nThat's the direction I want to continue exploring.\n\nTech Stack\n\nMurf Falcon — Text-to-Speech\n\nLiveKit Agents — Real-time voice infrastructure\n\nDeepgram — Speech-to-Text\n\nGemini — LLM\n\nPython — Agent logic and orchestration\n\nProject Links\n\n🔗 GitHub: [YOUR PUBLIC GITHUB REPOSITORY]\n\n🎥 Demo: [YOUR DEMO LINK]\n\n💼 LinkedIn: [YOUR LINKEDIN PROFILE]", "url": "https://wpnews.pro/news/ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response", "canonical_source": "https://dev.to/asmi_raut/building-ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response-20pc", "published_at": "2026-08-15 17:41:20+00:00", "updated_at": "2026-08-15 18:12:26.862322+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "developer-tools"], "entities": ["Ashraya AI", "Murf AI", "LiveKit Agents", "Deepgram", "Gemini", "Murf Falcon"], "alternates": {"html": "https://wpnews.pro/news/ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response", "markdown": "https://wpnews.pro/news/ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response.md", "text": "https://wpnews.pro/news/ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response.txt", "jsonld": "https://wpnews.pro/news/ashraya-ai-my-10-day-journey-building-a-voice-agent-for-disaster-response.jsonld"}}