Hi, I'm Gowtham M.
Over the last 10 days, I participated in the 10 Days of Voice Agents — VoiceForBharat Edition and built MoneyBuddy, an AI-powered voice assistant designed to make financial and government scheme-related information more accessible through natural conversations.
What started as a basic voice interaction gradually evolved into a system involving real-time AI, voice processing, memory, tool usage, human escalation, analytics, and multi-agent handoffs.
The goal was simple:
Can users get help with financial and government scheme-related questions through a natural voice conversation instead of navigating multiple websites and forms?
What started as a basic voice interaction gradually evolved into a system involving real-time AI, voice processing, memory, tool usage, human escalation, analytics, and multi-agent handoffs.
Users often need answers to questions like:
Instead of searching through multiple sources, MoneyBuddy is designed to let users speak naturally to an AI assistant.
The core flow looks like this:
User Voice
↓
Speech-to-Text
↓
MoneyBuddy AI Agent
↓
Tools / Memory / Agent Routing
↓
Response Generation
↓
Text-to-Speech
↓
User
I started by understanding how a real-time voice agent works.
The basic pipeline was:
Voice Input → AI Processing → Voice Output
This became the foundation for MoneyBuddy.
A voice assistant cannot simply read long chatbot-style responses.
I focused on making interactions:
This made me think more about user experience, not just AI responses.
I worked on giving MoneyBuddy clear instructions.
The agent needed to understand:
One important lesson: an AI agent performs better when its responsibilities are clearly defined.
Some information should not be generated or guessed by an LLM.
So I explored structured tool usage.
User Request
↓
Agent Understands Intent
↓
Tool Required?
↓
Call Tool
↓
Get Result
↓
Explain Result to User
This was an important shift from a simple conversational AI system toward an AI agent capable of taking structured actions.
I explored how MoneyBuddy could remember useful conversation context.
The goal was to avoid making returning users repeat the same information.
At the same time, memory requires responsible handling.
The agent should ask for consent before storing relevant information.
This taught me that AI memory is not just a database feature—it is also a user trust feature.
Not every problem should be handled completely by AI.
For situations requiring additional help or manual intervention, I explored human escalation.
A useful AI system should not pretend to know everything.
Sometimes the best response is knowing when to say:
This requires additional assistance.
My key learning:
Good AI systems should understand their limitations.
At this stage, I started looking at MoneyBuddy as a complete system.
The focus moved toward:
This was when the project started becoming more than a basic voice demo.
Building an agent is only the first step.
I also explored tracking what happens during interactions.
Examples of useful metrics include:
This helped me understand an important engineering principle:
If you cannot observe a system, it becomes difficult to improve it.
This was one of the biggest upgrades to MoneyBuddy.
Instead of asking one agent to handle everything, I added a dedicated Government Scheme Specialist.
The architecture became:
User
↓
MoneyBuddy Main Agent
↓
Is specialist help needed?
↙ ↘
No Yes
↓ ↓
Main Agent Scheme Specialist
↘ ↙
Final Response
↓
User
The main agent handles:
The specialist handles:
The important part was context handoff.
The user should not have to explain the entire problem again after being transferred.
This was my biggest takeaway from multi-agent systems:
Specialized agents with clear responsibilities can be easier to manage than one agent trying to do everything.
By the end of the challenge, MoneyBuddy had evolved into a system involving:
Voice AI
+
LLM
+
Clear Instructions
+
Structured Tools
+
Memory
+
Consent
+
Human Escalation
+
Analytics
+
Specialist Routing
+
Multi-Agent Handoff
Building MoneyBuddy taught me that a voice AI application is much more than:
LLM + Microphone + Speaker
A useful AI agent needs:
Some of the biggest challenges were:
A voice pipeline contains multiple stages:
Speech → STT → LLM → Tools → TTS → Audio
A delay at any stage affects the overall user experience.
Transferring a conversation between agents requires clear routing logic and context management.
Without proper context transfer, users may need to repeat themselves.
As the system grew, it became increasingly important to clearly define what each agent should handle.
I would like to continue improving the project by exploring:
These 10 days changed the way I think about AI applications.
I started with the idea of building a voice assistant.
I ended up learning about real-time systems, AI agents, tools, memory, user consent, escalation, observability, and multi-agent architectures.
My biggest takeaway:
A useful AI product is not built by choosing only a powerful model. It is built by connecting the right components with clear responsibilities, good user experience, reliable systems, and continuous testing.
MoneyBuddy is still a work in progress, but these 10 days gave me a much stronger foundation for building AI agents that can interact with users in more natural and useful ways.