{"slug": "building-a-voice-agent-in-10-days-my-voiceforbharat-journey", "title": "Building a Voice Agent in 10 Days — My VoiceForBharat Journey", "summary": "A developer built a multilingual voice assistant for a local Indian grocery store as part of the 10 Days of Voice Agents — VoiceForBharat Edition challenge. The system uses Deepgram for speech-to-text, Google Gemini for language understanding, Murf Falcon for text-to-speech, and LiveKit for real-time audio, with features like caller memory, safety guardrails, and specialist handoffs. The project demonstrates how to turn a basic voice agent into a practical customer-service system.", "body_md": "Building a voice agent sounds simple at first.\n\nListen to the user, send the text to an LLM, generate a response, and speak it back.\n\nBut once you start adding real-world requirements — memory, safety guardrails, multilingual conversations, phone calls, human escalation, analytics, and specialist handoffs — it becomes a very different engineering problem.\n\nOver the last 10 days, I worked on exactly that as part of **10 Days of Voice Agents — VoiceForBharat Edition**. I started with a basic voice agent and gradually turned it into a voice assistant designed for a local Indian store.\n\nHere's what I built and what I learned along the way.\n\nFor a local grocery or general store, customers often have simple questions:\n\nThese questions don't always need a person to answer them. I wanted to build a voice-first assistant that could handle these conversations naturally, while also knowing when it should stop trying to solve the problem itself.\n\nVoice is particularly useful here because customers don't have to open an app, type their question, or navigate through menus. They can simply speak.\n\nMy project is a voice assistant for a local Indian grocery/general store. It's designed to:\n\nThe goal wasn't just to make an AI that could talk. It was to make the conversation behave like a useful customer-service system.\n\nThe basic voice pipeline looks like this:\n\n```\nUser speaks\n    ↓\nDeepgram Speech-to-Text\n    ↓\nGoogle Gemini LLM\n    ↓\nTools / Memory / Agent Logic\n    ↓\nMurf Falcon TTS\n    ↓\nLiveKit real-time audio\n    ↓\nUser hears the response\n```\n\nThe agent session also uses multilingual turn detection, voice activity detection, preemptive generation, and noise cancellation to make the conversation feel more natural.\n\nOne of the most important parts of this project was the voice itself.\n\nI used Murf Falcon with an Indian English voice and configured the agent to switch between `en-IN`\n\nand `hi-IN`\n\ndepending on the conversation. The project detects Hindi/Hinglish-style input and adjusts the TTS locale accordingly, so the interaction isn't restricted to formal English — a user can speak in English, Hindi, or Hinglish, and the agent responds in the same conversational style.\n\nFor this challenge, I used Murf Falcon — the fastest TTS API — as the text-to-speech layer of the agent.\n\nA useful assistant shouldn't treat every conversation as if it's meeting the customer for the first time.\n\nI added caller memory using SQLite. The agent has two tools:\n\n```\nlookup_caller()\nsave_caller_memory()\n```\n\nWhen a caller is recognized, the agent looks up previously saved information. If the customer shares useful preferences, the agent asks for permission before saving them.\n\nThis was an important design decision — the agent shouldn't silently collect information just because it can. The system prompt explicitly prevents the agent from inventing memories or exposing internal database information.\n\nAnother important part of the project was defining what the agent should and shouldn't do. The assistant has clear objectives and guardrails. For example, it should never:\n\nIf it doesn't know something, it's instructed to say so instead of guessing. This might seem simple, but good voice agents need clear boundaries just as much as they need good prompts.\n\nOne of the features I enjoyed building was the specialist handoff.\n\nThe main store assistant handles normal shopping questions. But if the customer needs help with returns, refunds, damaged or defective products, wrong or missing items, or return eligibility, the main agent transfers the conversation to a dedicated **Returns and Refunds Specialist**.\n\nThe important part is that the specialist receives the existing conversation context — the customer doesn't have to explain the problem again. The flow looks like this:\n\n```\nCustomer\n   ↓\nMain Store Assistant\n   ↓\nReturn / Refund request detected\n   ↓\nReturns & Refunds Specialist\n   ↓\nContinue existing conversation\n```\n\nThis made the project feel much closer to a real customer-support system than a single chatbot.\n\nI also added outbound calling using LiveKit's SIP capabilities. The outbound call flow creates a unique room, dispatches the agent to that room, and creates an outbound SIP participant:\n\n```\nAgent\n  ↓\nLiveKit Room\n  ↓\nSIP Participant\n  ↓\nPhone Network\n  ↓\nCustomer\n```\n\nThis was one of the more challenging parts, since there are more moving pieces here than in a browser-based voice conversation. The system has to coordinate the LiveKit agent, the room, the SIP configuration, and the phone connection.\n\nOnce an agent starts making calls, another question appears: **how do I know whether it's actually performing well?**\n\nSo I added call tracking and analytics. Each call can record:\n\nThe backend exposes this data through a FastAPI service, which provides metrics like total calls, successful calls, failed calls, escalated calls, success rate, and average duration. It also supports filtering and retrieving individual call details.\n\nThis changed how I looked at the project. Instead of only asking, \"Does the agent talk?\" I could start asking, \"Did the conversation accomplish its goal?\"\n\nThe hardest part wasn't getting the first response from the agent — it was making all the pieces work together.\n\n```\nFrontend\n   ↓\nLiveKit\n   ↓\nVoice Agent\n   ├── Speech-to-Text\n   ├── LLM\n   ├── Text-to-Speech\n   ├── Tools\n   ├── Memory\n   ├── SIP\n   └── Analytics\n```\n\nWhen something goes wrong, it isn't always obvious which layer is responsible. During the challenge, I dealt with issues around running multiple services, environment variables, database state, call handling, and getting components to communicate correctly.\n\nOne lesson I learned: debugging a voice agent isn't just about debugging Python code. You also have to think about audio flow, real-time connections, API credentials, room state, SIP state, database state, and agent state.\n\nBreaking the system into smaller components and checking each layer separately made troubleshooting much easier.\n\nYou don't need all of these features on day one. Start with the basic pipeline:\n\n```\nSpeech-to-Text\n      ↓\n     LLM\n      ↓\nText-to-Speech\n```\n\nThen add real-time transport such as LiveKit. From there, gradually layer in:\n\nMy project is available publicly on GitHub: [Codehunter0009/murf-livekit-starter](https://github.com/Codehunter0009/murf-livekit-starter)\n\nYou can inspect the complete implementation, including the backend, frontend, agent logic, analytics API, database, and outbound calling code.\n\nThe project uses Python for the backend and Node.js for the frontend. The backend uses `uv`\n\nfor dependency management.\n\nAfter cloning the repository:\n\n```\ncd backend\nuv sync\n```\n\nThen configure the required environment variables in `.env.local`\n\n:\n\n```\nLIVEKIT_URL\nLIVEKIT_API_KEY\nLIVEKIT_API_SECRET\nMURF_API_KEY\nDEEPGRAM_API_KEY\nGOOGLE_API_KEY\n```\n\nFor outbound calling, the relevant SIP configuration is also kept in environment variables.\n\nNever commit your`.env.local`\n\nfile or API keys to GitHub.\n\nStart the backend in development mode:\n\n```\nuv run python src/agent.py dev\n```\n\nThe project also includes a frontend for interacting with the voice agent through the browser. Once the backend, LiveKit, and frontend are running, open the application, allow microphone access, and start a conversation.\n\nThe biggest lesson from these 10 days is that building a voice agent is much more than connecting an LLM to a microphone. A useful voice agent needs:\n\n**Conversation + Context + Tools + Safety + Observability**\n\nThe LLM is only one part of the system. I also learned how important it is to design failure paths:\n\nThese are the questions that turn a demo into a complete system.\n\nThere's plenty I'd still improve: more robust multilingual support, an expanded toolset, a better analytics dashboard, additional specialist agents, and a more reliable phone experience. I'd also like to spend more time measuring latency and real-world conversation quality, rather than evaluating the system only through individual test calls.\n\n**GitHub:** [Codehunter0009/murf-livekit-starter](https://github.com/Codehunter0009/murf-livekit-starter)\n\nThe complete source code and setup instructions are available in the repository.\n\nTen days ago, I was mainly thinking about how to make a voice agent talk.\n\nBy the end of the challenge, I was thinking about something much bigger: How should a voice agent behave when it doesn't know something? How should it remember users? When should it ask for permission? When should it escalate? How do we measure whether a call was successful? And how do we make the whole system reliable enough for someone to actually use?\n\nThat's what made this challenge valuable. I didn't just learn how to build a voice interface — I learned how to think about a voice agent as a complete system.\n\n**10 Days of Voice Agents — VoiceForBharat Edition completed. 🚀**", "url": "https://wpnews.pro/news/building-a-voice-agent-in-10-days-my-voiceforbharat-journey", "canonical_source": "https://dev.to/nittala_koushik_3b5521b57/building-a-voice-agent-in-10-days-my-voiceforbharat-journey-2abh", "published_at": "2026-08-15 06:03:19+00:00", "updated_at": "2026-08-15 06:11:06.140929+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "natural-language-processing", "developer-tools"], "entities": ["Deepgram", "Google Gemini", "Murf Falcon", "LiveKit", "VoiceForBharat", "SQLite"], "alternates": {"html": "https://wpnews.pro/news/building-a-voice-agent-in-10-days-my-voiceforbharat-journey", "markdown": "https://wpnews.pro/news/building-a-voice-agent-in-10-days-my-voiceforbharat-journey.md", "text": "https://wpnews.pro/news/building-a-voice-agent-in-10-days-my-voiceforbharat-journey.txt", "jsonld": "https://wpnews.pro/news/building-a-voice-agent-in-10-days-my-voiceforbharat-journey.jsonld"}}