# Ashraya AI: My 10-Day Journey Building a Voice Agent for Disaster Response > Source: > Published: 2026-08-15 17:41:20+00:00 From a simple voice assistant to a multi-agent disaster-response system Disasters don't wait for people to open an app, read instructions, or type a message. In an emergency, people need information quickly—and sometimes they simply need to talk. That was the idea behind Ashraya AI, the disaster-response voice agent I built during the 10 Days of Voice Agents — VoiceForBharat Edition challenge by Murf AI. Over the past 10 days, I went from building a basic conversational voice agent to developing a system with memory, tools, outbound calling, call analytics, success tracking, human escalation, and specialist-agent handoffs. The goal was not to build another chatbot that can answer everything. The goal was to build a voice-first system that can guide people during disaster situations while being explicit about what it does not know. **1. The Problem** During disasters such as floods, earthquakes, cyclones, or other emergencies, information can become difficult to access. A person may need answers to questions like: What should I do right now? What should I take if I need to leave? Where can I get help? How should I prepare an elderly family member for evacuation? Can someone help me if I cannot handle the situation myself? A traditional text interface isn't always ideal in these situations. Someone may be frightened, busy helping family members, unable to type comfortably, or simply need to communicate naturally. That's where a voice interface can be useful. Ashraya AI is designed as a conversational disaster-response assistant that users can interact with through voice. The project focuses on providing practical guidance while maintaining strict safety boundaries. **2. Why Voice?** Voice changes the interaction model. Instead of navigating menus or typing questions, a user can simply say: "There is flooding near my house. What should I do?" or: "I may need to go to a shelter. What should I take?" The agent can respond conversationally and ask relevant follow-up questions. For a disaster-response use case, this matters because the user's attention may already be occupied by the emergency itself. But voice introduces another challenge: An AI that sounds confident can easily sound authoritative—even when it is wrong. That made safety and hallucination prevention one of the most important parts of this project. **3. What is Ashraya AI?** Ashraya AI is a voice-based disaster-response assistant built using: Murf Falcon — text-to-speech LiveKit Agents — real-time voice-agent infrastructure Deepgram — speech-to-text Gemini — language model Python — application logic and agent orchestration The architecture evolved throughout the challenge. Instead of treating Ashraya as one giant agent that handles every possible question, I eventually introduced a specialist-agent architecture. The main agent handles the overall disaster-response conversation. When the conversation becomes specifically shelter-related, it can hand the conversation to a Shelter Information Specialist. This creates a much clearer separation of responsibilities. **4. System Architecture** At a high level, the voice interaction works like this: ``` User │ ▼ Microphone │ ▼ Speech-to-Text (Deepgram) │ ▼ Main AI Agent (Gemini) │ ┌───────────┴───────────┐ │ │ ▼ ▼ General disaster Shelter-related assistance request │ ▼ Shelter Information Specialist │ ▼ Shelter Guidance (without fabricated live information) │ ▼ Murf Falcon TTS │ ▼ User ``` The important design decision here is that the specialist is not simply another prompt inside the main agent. It has its own instructions, scope, limitations, and success tracking. **5. Building the Voice Agent** A real-time voice agent fundamentally needs several components working together: Speech-to-Text The user's speech needs to be converted into text so the language model can understand it. I used Deepgram for speech recognition. Language Model The language model handles reasoning, conversation, tool selection, and responses. I used Gemini for this layer. Text-to-Speech The generated response needs to be converted back into natural speech. For this project, I used Murf Falcon. Real-Time Transport The system needs a real-time communication layer to move audio between the user and the agent. I used LiveKit Agents. Together, these components form the basic voice-agent pipeline: User Speech ↓ Speech-to-Text ↓ LLM / Agent Logic ↓ Text-to-Speech ↓ Spoken Response **6. Giving the Agent a Clear Personality and Safety Boundary** One of the first lessons I learned was that an agent's prompt is not just about telling it what to say. It also needs to define: What it is allowed to do What it is not allowed to do When it should ask questions When it should use a tool When it should escalate What information it must never invent For a disaster-response system, this becomes especially important. For example, an unsafe agent might respond to: "Which government shelter is open near me?" with a fabricated answer such as: "The nearest shelter is XYZ School." That would be unacceptable. Ashraya AI is explicitly instructed not to invent live shelter locations, availability, capacity, evacuation routes, or government orders when it doesn't have verified real-time data. Instead, it should clearly state the limitation and direct the caller toward appropriate official local authorities. That distinction became one of the most important safety principles in the project: Not knowing something is better than confidently making something up. **7. Memory for Returning Users** Another part of the project was giving Ashraya AI memory. The idea was to avoid treating every conversation as if it were happening with a completely new user. Memory can allow the system to retain useful conversational context and provide more consistent interactions when appropriate. However, memory in a disaster-response system also needs to be handled carefully. The agent should use relevant context without making assumptions about the user's current emergency. Current information from the caller should always take priority over stale conversational context. **8. Tools and Actions** The agent is not limited to generating text. Tools allow it to perform specific actions. This is important because an AI agent should not rely on the language model to simulate actions that should actually happen in software. For example, instead of simply saying: "I'll connect you to a specialist." the system can invoke a handoff tool that actually transfers the conversation. This became particularly important during Day 9. **9. Multi-Agent Handoff: The Shelter Specialist** One of the biggest architectural changes was introducing a dedicated Shelter Information Specialist. The main Ashraya AI agent is broader. The shelter specialist is intentionally narrow. It focuses on questions such as: What should I take to a shelter? How should I prepare before moving? What documents should I carry? What medicines or essential supplies should I take? What should I consider when travelling with children or older adults? What information should I obtain from local authorities? But it is not a general disaster-response agent. If someone asks the specialist: "What should I do during an earthquake?" or: "How do I perform CPR?" the specialist should not suddenly become a medical or earthquake-response expert. It should say that the question is outside its role and redirect the user appropriately. This is a subtle but important difference between a multi-agent system and a single agent with a very large prompt. The goal isn't: "Make every agent know everything." The goal is: "Give each agent a clear responsibility and let the system decide who should handle the conversation." **10. Preventing Hallucinated Shelter Information** This was one of the most important safety challenges in the project. Suppose someone asks: "Which government shelter is open near me right now?" If the system doesn't have a verified live shelter database, it cannot honestly answer that question. So the Shelter Information Specialist explicitly states that it does not have verified real-time shelter-location or opening-status information. It can instead explain how the caller can obtain verified information from appropriate local authorities. This is an important design principle for real-world AI systems: The absence of information should produce a limitation—not a hallucination. A disaster-response assistant should be conservative when dealing with information that can directly affect someone's physical safety. **11. Outbound Calling** Ashraya AI was also designed to support outbound voice interactions. This changes the experience from: "A user opens an AI application." to: "The AI can initiate a voice conversation." For a disaster-response scenario, outbound communication could eventually be useful for alerting people, checking whether they need assistance, or providing information. However, outbound calling also introduces additional considerations around consent, privacy, call reliability, and responsible use. **12. Human Escalation** Another important capability was allowing the system to recognize when AI should not be the final layer of support. There are situations where a human operator or emergency service may be more appropriate. The principle is simple: AI should assist humans, not pretend to replace emergency infrastructure. The agent should never claim that emergency personnel are already responding unless the system actually knows that. **13. Call Analytics and Success Tracking** Another lesson from the project was that a conversation isn't necessarily successful simply because the AI produced a response. I added call outcome tracking so conversations can be evaluated based on whether the intended objective was actually completed. For example, a shelter-related interaction should only be marked successful after meaningful shelter guidance has actually been provided. A transfer alone should not automatically count as success. This distinction is important for evaluating voice agents in production. You need to measure outcomes, not just conversations. **14. The Hardest Part: Making the Agent Stay in Its Lane** One of the biggest challenges I encountered was getting the specialist-agent architecture to behave correctly. At one point, the Shelter Information Specialist was still answering questions about earthquakes and CPR. That was a clear architectural failure. The specialist had been created, but its boundaries were not being enforced strongly enough. I tightened its instructions so that it explicitly refuses questions outside its shelter-information scope. This taught me an important lesson: Creating a specialist agent isn't enough. You have to define its boundaries as carefully as its capabilities. **15. Debugging the Handoff** Another issue I encountered was a technical handoff failure. The main agent successfully called the shelter-transfer tool, but the specialist failed to initialize because of an incorrect prompt variable reference. The result was a NameError during the transfer. The conversation looked correct from the user's perspective until the actual handoff happened—but internally, the specialist could not be created. The problem wasn't with the AI reasoning. It was a straightforward software integration issue. That distinction matters when debugging agentic systems. Sometimes the model isn't the problem. Your orchestration code is. After correcting the specialist initialization and prompt reference, the handoff could work as intended. **16. Another Important Lesson: "Transfer" Must Actually Transfer** One of the most interesting things I learned was that saying: "I'm connecting you to the Shelter Information Specialist." doesn't mean the transfer actually happened. The application needs to: Detect the appropriate intent. Invoke the transfer tool. Create or activate the specialist agent. Pass the relevant conversation context. Return control to the specialist. Ensure the specialist follows its own instructions. If any of these steps fail, the main agent may continue speaking. That creates a confusing experience where the user asks: "Who am I speaking to?" and the system gives inconsistent answers. This was a useful reminder that agent handoffs are software orchestration problems as much as they are prompting problems. **17. How to Run the Project** The project is built with Python and LiveKit. After cloning the repository, install the required dependencies and configure the required environment variables. The API keys should be stored in an environment file such as: ``` .env ``` and should never be committed to GitHub. A typical setup looks like: ``` git clone https://github.com/AsmiRaut23/murf-livekit-starter cd murf-livekit-starter python -m venv .venv ``` Activate the environment and install the project's dependencies: ``` pip install -r requirements.txt ``` Then configure the required credentials in your environment variables. For example: ``` LIVEKIT_URL=... LIVEKIT_API_KEY=... LIVEKIT_API_SECRET=... MURF_API_KEY=... DEEPGRAM_API_KEY=... GEMINI_API_KEY=... ``` The exact variables depend on the project's implementation. Once the environment is configured, start the LiveKit agent using the project's entry point. Do not publish real API keys, phone numbers, caller information, or other private credentials in the repository or blog. **18. Testing the Agent** The easiest way to test a voice agent is through real conversations rather than only checking whether the code runs. For Ashraya AI, I tested scenarios such as: General disaster question "There is flooding near my house. What should I do?" Shelter question "I may need to go to a shelter. What should I take?" Live-information limitation "Which government shelter is open near me right now?" Specialist boundary After transferring to the Shelter Information Specialist: "What should I take to the shelter?" and then testing an unrelated question such as: "How do I perform CPR?" The specialist should not start behaving like a general medical assistant. Testing these boundary cases was just as important as testing successful conversations. **19. What I Would Improve Next** Ashraya AI is still a prototype, and there are several areas I would improve before treating it as a production disaster-response system. Verified real-time shelter data The biggest improvement would be connecting the shelter specialist to an authoritative, continuously updated government or disaster-management data source. That would allow the system to answer location-specific shelter questions based on verified information rather than simply explaining that live data is unavailable. Better emergency escalation A production version would need carefully designed integration with emergency services and human operators. More robust multilingual support India has enormous linguistic diversity. Expanding reliable multilingual and code-mixed conversations would make the system more accessible. Stronger evaluation I would also build a larger evaluation suite covering: Hallucination resistance Specialist boundaries Emergency scenarios Tool failures Handoff failures Call completion Latency False transfers Successful outcomes **20. What I Learned in 10 Days** The biggest lesson from this challenge wasn't how to connect speech-to-text to an LLM and then to text-to-speech. It was learning that a useful voice agent is much more than a model that can talk. A reliable system needs: Conversation + tools + memory + guardrails + orchestration + observability + failure handling. And sometimes the most important behavior of an AI agent is knowing when not to answer. Building Ashraya AI also changed how I think about multi-agent systems. I initially thought the challenge was to make one agent increasingly capable. By the end, I realized that a better approach is often to build a system of focused agents with clear responsibilities. Final Thoughts Over these 10 days, Ashraya AI evolved from a voice assistant into a more structured disaster-response system. It can: Have natural voice conversations Use an Indian voice powered by Murf Falcon Follow explicit safety guardrails Maintain conversational context Use tools Support outbound calling Track call outcomes Escalate when necessary Hand shelter-related conversations to a specialist agent Refuse to fabricate information it cannot verify The project is still a work in progress. And that's probably the most valuable part of the experience. Building a voice agent isn't about making AI sound impressive. It's about making the system useful, predictable, and honest about its limitations. That's the direction I want to continue exploring. Tech Stack Murf Falcon — Text-to-Speech LiveKit Agents — Real-time voice infrastructure Deepgram — Speech-to-Text Gemini — LLM Python — Agent logic and orchestration Project Links 🔗 GitHub: [YOUR PUBLIC GITHUB REPOSITORY] 🎥 Demo: [YOUR DEMO LINK] 💼 LinkedIn: [YOUR LINKEDIN PROFILE]