# Building a Voice Agent in 10 Days — My VoiceForBharat Journey > Source: > Published: 2026-08-15 06:03:19+00:00 Building a voice agent sounds simple at first. Listen to the user, send the text to an LLM, generate a response, and speak it back. But once you start adding real-world requirements — memory, safety guardrails, multilingual conversations, phone calls, human escalation, analytics, and specialist handoffs — it becomes a very different engineering problem. Over the last 10 days, I worked on exactly that as part of **10 Days of Voice Agents — VoiceForBharat Edition**. I started with a basic voice agent and gradually turned it into a voice assistant designed for a local Indian store. Here's what I built and what I learned along the way. For a local grocery or general store, customers often have simple questions: These questions don't always need a person to answer them. I wanted to build a voice-first assistant that could handle these conversations naturally, while also knowing when it should stop trying to solve the problem itself. Voice is particularly useful here because customers don't have to open an app, type their question, or navigate through menus. They can simply speak. My project is a voice assistant for a local Indian grocery/general store. It's designed to: The goal wasn't just to make an AI that could talk. It was to make the conversation behave like a useful customer-service system. The basic voice pipeline looks like this: ``` User speaks ↓ Deepgram Speech-to-Text ↓ Google Gemini LLM ↓ Tools / Memory / Agent Logic ↓ Murf Falcon TTS ↓ LiveKit real-time audio ↓ User hears the response ``` The agent session also uses multilingual turn detection, voice activity detection, preemptive generation, and noise cancellation to make the conversation feel more natural. One of the most important parts of this project was the voice itself. I used Murf Falcon with an Indian English voice and configured the agent to switch between `en-IN` and `hi-IN` depending on the conversation. The project detects Hindi/Hinglish-style input and adjusts the TTS locale accordingly, so the interaction isn't restricted to formal English — a user can speak in English, Hindi, or Hinglish, and the agent responds in the same conversational style. For this challenge, I used Murf Falcon — the fastest TTS API — as the text-to-speech layer of the agent. A useful assistant shouldn't treat every conversation as if it's meeting the customer for the first time. I added caller memory using SQLite. The agent has two tools: ``` lookup_caller() save_caller_memory() ``` When a caller is recognized, the agent looks up previously saved information. If the customer shares useful preferences, the agent asks for permission before saving them. This was an important design decision — the agent shouldn't silently collect information just because it can. The system prompt explicitly prevents the agent from inventing memories or exposing internal database information. Another important part of the project was defining what the agent should and shouldn't do. The assistant has clear objectives and guardrails. For example, it should never: If it doesn't know something, it's instructed to say so instead of guessing. This might seem simple, but good voice agents need clear boundaries just as much as they need good prompts. One of the features I enjoyed building was the specialist handoff. The main store assistant handles normal shopping questions. But if the customer needs help with returns, refunds, damaged or defective products, wrong or missing items, or return eligibility, the main agent transfers the conversation to a dedicated **Returns and Refunds Specialist**. The important part is that the specialist receives the existing conversation context — the customer doesn't have to explain the problem again. The flow looks like this: ``` Customer ↓ Main Store Assistant ↓ Return / Refund request detected ↓ Returns & Refunds Specialist ↓ Continue existing conversation ``` This made the project feel much closer to a real customer-support system than a single chatbot. I also added outbound calling using LiveKit's SIP capabilities. The outbound call flow creates a unique room, dispatches the agent to that room, and creates an outbound SIP participant: ``` Agent ↓ LiveKit Room ↓ SIP Participant ↓ Phone Network ↓ Customer ``` This was one of the more challenging parts, since there are more moving pieces here than in a browser-based voice conversation. The system has to coordinate the LiveKit agent, the room, the SIP configuration, and the phone connection. Once an agent starts making calls, another question appears: **how do I know whether it's actually performing well?** So I added call tracking and analytics. Each call can record: The backend exposes this data through a FastAPI service, which provides metrics like total calls, successful calls, failed calls, escalated calls, success rate, and average duration. It also supports filtering and retrieving individual call details. This changed how I looked at the project. Instead of only asking, "Does the agent talk?" I could start asking, "Did the conversation accomplish its goal?" The hardest part wasn't getting the first response from the agent — it was making all the pieces work together. ``` Frontend ↓ LiveKit ↓ Voice Agent ├── Speech-to-Text ├── LLM ├── Text-to-Speech ├── Tools ├── Memory ├── SIP └── Analytics ``` When something goes wrong, it isn't always obvious which layer is responsible. During the challenge, I dealt with issues around running multiple services, environment variables, database state, call handling, and getting components to communicate correctly. One lesson I learned: debugging a voice agent isn't just about debugging Python code. You also have to think about audio flow, real-time connections, API credentials, room state, SIP state, database state, and agent state. Breaking the system into smaller components and checking each layer separately made troubleshooting much easier. You don't need all of these features on day one. Start with the basic pipeline: ``` Speech-to-Text ↓ LLM ↓ Text-to-Speech ``` Then add real-time transport such as LiveKit. From there, gradually layer in: My project is available publicly on GitHub: [Codehunter0009/murf-livekit-starter](https://github.com/Codehunter0009/murf-livekit-starter) You can inspect the complete implementation, including the backend, frontend, agent logic, analytics API, database, and outbound calling code. The project uses Python for the backend and Node.js for the frontend. The backend uses `uv` for dependency management. After cloning the repository: ``` cd backend uv sync ``` Then configure the required environment variables in `.env.local` : ``` LIVEKIT_URL LIVEKIT_API_KEY LIVEKIT_API_SECRET MURF_API_KEY DEEPGRAM_API_KEY GOOGLE_API_KEY ``` For outbound calling, the relevant SIP configuration is also kept in environment variables. Never commit your`.env.local` file or API keys to GitHub. Start the backend in development mode: ``` uv run python src/agent.py dev ``` The project also includes a frontend for interacting with the voice agent through the browser. Once the backend, LiveKit, and frontend are running, open the application, allow microphone access, and start a conversation. The biggest lesson from these 10 days is that building a voice agent is much more than connecting an LLM to a microphone. A useful voice agent needs: **Conversation + Context + Tools + Safety + Observability** The LLM is only one part of the system. I also learned how important it is to design failure paths: These are the questions that turn a demo into a complete system. There's plenty I'd still improve: more robust multilingual support, an expanded toolset, a better analytics dashboard, additional specialist agents, and a more reliable phone experience. I'd also like to spend more time measuring latency and real-world conversation quality, rather than evaluating the system only through individual test calls. **GitHub:** [Codehunter0009/murf-livekit-starter](https://github.com/Codehunter0009/murf-livekit-starter) The complete source code and setup instructions are available in the repository. Ten days ago, I was mainly thinking about how to make a voice agent talk. By the end of the challenge, I was thinking about something much bigger: How should a voice agent behave when it doesn't know something? How should it remember users? When should it ask for permission? When should it escalate? How do we measure whether a call was successful? And how do we make the whole system reliable enough for someone to actually use? That's what made this challenge valuable. I didn't just learn how to build a voice interface — I learned how to think about a voice agent as a complete system. **10 Days of Voice Agents — VoiceForBharat Edition completed. 🚀**