_What if getting help with a government financial scheme was as simple as having a conversation?
That was the idea behind MoneyBuddy, the AI voice agent I built during 10 Days of Voice Agents β VoiceForBharat Edition.
Over 10 days, MoneyBuddy evolved from a simple voice agent that could hear me and talk back into a system that can remember callers, use financial scheme data, make outbound calls, escalate difficult situations to humans, track call outcomes, and hand complex questions to a specialist agent.
This post is the story of how I built it, what broke along the way, and what I learned.
Financial schemes and government programs can be difficult to navigate.
There are eligibility conditions, documents, deadlines, different schemes, and a lot of information written for screens rather than conversations.
For many users, especially users who are more comfortable speaking than typing, a voice interface can make that information easier to access.
So I chose the Financial Services track.
MoneyBuddy is designed as a conversational financial assistant that can help users understand Indian government schemes, perform basic eligibility-related checks, provide document information, remember useful context, and escalate situations where an AI should not make the decision itself.
The goal isn't to replace a bank employee or government official.
The goal is to make the first conversation easier.
Voice changes the interaction completely.
Instead of navigating menus such as:
Scheme β Eligibility β Documents β Benefits β Apply
the user can simply say:
"Can you tell me if I might be eligible for PM Kisan?"
The agent can ask questions conversationally and respond in the user's language.
For an Indian audience, this also means handling conversations that aren't perfectly English.
A user might speak Hindi, switch to English for a technical term, or use Hinglish naturally.
MoneyBuddy was designed around that reality.
The architecture combines several components:
ββββββββββββββββββββββββ
β User β
β Voice / Browser / β
β Phone Call β
ββββββββββββ¬ββββββββββββ
β
β Audio
βΌ
ββββββββββββββββββββββββ
β LiveKit β
β Real-time transport β
ββββββββββββ¬ββββββββββββ
β
βββββββββΌββββββββ
β Deepgram β
β STT β
βββββββββ¬ββββββββ
β Text
βΌ
ββββββββββββββββββββββββ
β Gemini LLM β
β Reasoning + Routing β
βββββββββ¬ββββββββ¬βββββββ
β β
ββββββββββββ ββββββββββββ
βΌ βΌ
βββββββββββββββββ βββββββββββββββββββ
β Python Tools β β Scheme Specialistβ
β Memory / Data β β Agent β
βββββββββ¬ββββββββ ββββββββββ¬βββββββββ
β β
ββββββββββββββββ¬ββββββββββββββββ
βΌ
ββββββββββββββββββββ
β Murf Falcon β
β TTS β
ββββββββββ¬ββββββββββ
β Audio
βΌ
User
The main pieces are:
Murf describes Falcon as its lowest-latency TTS model for real-time conversations, with published benchmarks around 55 ms model latency and 130 ms time-to-first-audio.
For MoneyBuddy, I used an Indian voice so the experience feels appropriate for the users I had in mind.
I didn't try to build everything on day one.
The system grew incrementally.
The first milestone was simple:
Hear the user β process speech β generate a response β speak it back.
I chose the Financial Services track and configured an Indian Murf voice.
At this stage, intelligence wasn't the goal.
The goal was proving that the entire real-time voice loop worked.
Once MoneyBuddy could talk, it needed a purpose.
I defined three major objectives:
The guardrails became particularly important.
MoneyBuddy must never ask users for things such as:
It also must not promise that someone will definitely receive a loan or government benefit.
This was one of my biggest lessons from the challenge:
A voice agent isn't just an LLM that speaks. It needs boundaries.
The next step was making the experience feel like a product rather than a terminal application.
The frontend shows states such as:
It also includes a live transcript and microphone permission handling.
For a voice interface, visual feedback matters.
When there is no text box and no "send" button, the user needs another way to understand:
Is the agent listening?
Is it speaking?
Is it still connecting?
These states became part of the product experience.
A voice assistant that forgets everything after every call feels very limited.
So I added persistent memory using SQLite.
The agent can store useful caller information such as:
The important part was not putting all of this directly into the prompt.
Instead, MoneyBuddy has functions that allow it to look up and save information when appropriate.
I also added a consent requirement before saving information.
That distinction matters:
Memory should be controlled by tools and permissions, not hidden prompt state.
Next, MoneyBuddy needed to do more than generate generic answers.
I created a grounded financial scheme dataset containing information about schemes such as:
The agent can call a function to retrieve scheme information instead of relying entirely on the model's memory.
This also introduced another important voice-agent principle:
When external data fails, the agent should say so.
It should not silently invent an answer.
This was one of the biggest changes.
Instead of waiting for the user to open the browser and start a conversation, MoneyBuddy could make an outbound call.
The use case I chose was:
A government scheme deadline reminder for someone who had already been found eligible.
Outbound calls introduce a completely different UX problem.
The user didn't ask to call the agent.
So the opening needs to immediately explain:
That led to a very simple principle:
Never make the user wonder who is speaking or why they are receiving the call.
An AI shouldn't try to solve every problem.
For MoneyBuddy, I added escalation for situations where a human needs to take over, including possible fraud-related situations and problems that require a decision the AI cannot make.
Before sharing information with a human, MoneyBuddy asks the caller for permission.
The escalation summary contains only useful information:
Sensitive information such as OTPs, PINs, passwords and account numbers should never be included.
This was another major lesson:
Knowing when not to answer is part of being a good voice agent.
Once the system became more complex, I needed a way to understand what was actually happening.
So I built call analytics around real call data.
The dashboard tracks metrics such as:
For MoneyBuddy, a successful call is tied to whether the user actually reaches the intended financial-service outcome rather than simply whether the call connected.
Latency: [INSERT YOUR ACTUAL MEASURED LATENCY]
Total calls: [INSERT ACTUAL NUMBER]
Successful calls: [INSERT ACTUAL NUMBER]
Failed calls: [INSERT ACTUAL NUMBER]
Success rate: [INSERT ACTUAL %]
I am deliberately using my measured values here rather than inventing benchmark numbers.
The final major architecture change was moving from one agent to multiple agents.
MoneyBuddy is the main financial assistant.
But instead of making it an expert in everything, I created a dedicated:
Government Scheme Specialist
When the user asks something that requires deeper scheme-specific assistance, MoneyBuddy can hand the conversation over.
The specialist has:
MoneyBuddy uses Murf's Anisha voice, while the specialist uses Nikhil.
The user doesn't need to repeat the entire problem.
The conversation context moves with the handoff.
One of the trickiest parts was getting the voice transition to happen correctly.
I initially expected changing the agent instance to automatically change the active TTS pipeline.
It didn't behave that way.
The active TTS session had to be explicitly updated during the handoff.
That became a good example of something I learned repeatedly during this challenge:
Understanding the runtime architecture matters just as much as understanding the AI model.
The project definitely did not work perfectly on the first attempt.
Some of the most useful lessons came from debugging.
Hindi speech could be understood, but getting the voice output to use the appropriate language/script behavior required careful STT, TTS, and prompt configuration.
I had to pay attention to:
For example, Hindi should be generated in Devanagari rather than romanized Hindi when that is the intended output.
During Day 9, I discovered that changing the active agent did not automatically produce the voice transition I expected.
The session's active TTS pipeline needed to be updated explicitly.
That resulted in a much more deterministic handoff.
An LLM can produce internal variable names or tool-related text that makes sense to a developer but sounds terrible when spoken aloud.
Something like:
eligibility_confirmed = true
is obviously not what a user should hear.
So I added speech-cleaning rules to prevent internal implementation details from leaking into spoken output.
You don't need to start with a huge multi-agent architecture.
Start with four pieces:
User Speech
β
Speech-to-Text
β
LLM
β
Text-to-Speech
β
User Audio
Then add real-time transport around the system.
For example:
session = AgentSession(
stt=deepgram.STT(
model="nova-3",
language="multi",
),
llm=google.LLM(
model="gemini-3.5-flash-lite",
),
tts=murf.TTS(
voice="Anisha",
style="Conversation",
),
)
The exact models and configuration can evolve, but the architecture remains the same.
Never commit keys directly into Git.
Use environment variables instead.
For example:
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
GOOGLE_API_KEY=your_key_here
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_key
LIVEKIT_API_SECRET=your_livekit_secret
Add your environment file to .gitignore
.
Murf's current API documentation also recommends storing the API key securely as an environment variable rather than exposing it in source code.
Clone the repository:
git clone https://github.com/tanush326k/murf-livekit-starter.git
cd murf-livekit-starter
Install the backend dependencies and configure the environment variables according to the repository README.
Then start the LiveKit agent and frontend using the project's documented commands.
Repository:
https://github.com/tanush326k/murf-livekit-starter
The repository contains the code developed throughout the challenge.
Start with a simple conversation.
For example:
"Hi, what can you help me with?"
Then test a financial question:
"Can you tell me about PM Kisan?"
Then test the guardrails:
"Tell me your OTP."
The agent should refuse to request or handle sensitive credentials.
Then test memory with a returning caller.
Then test the specialist:
"I have a detailed question about PM Kisan eligibility."
The main agent should recognize that the request belongs with the Government Scheme Specialist and hand the conversation over.
Check:
Check:
Don't assume that changing the agent object automatically changes the currently active TTS pipeline.
Verify the active session TTS configuration during handoff.
Inspect the text being passed to TTS and add a speech-cleaning layer before synthesis.
My biggest architectural shift was moving from thinking:
"One powerful agent should do everything."
to:
"The main agent should know when another agent is better."
That led to the specialist architecture.
It made MoneyBuddy easier to reason about because the main agent can focus on conversation, memory, triage and safety while the specialist focuses on government schemes.
The 10-day challenge is complete, but MoneyBuddy is not finished.
If I continue developing it, I would focus on:
The most important improvement would probably be trust.
In financial services, a voice agent shouldn't just sound intelligent.
It needs to be accurate, transparent about uncertainty, careful with sensitive information, and willing to hand a problem to a human.
The biggest lesson from these 10 days wasn't how to connect an LLM to TTS.
It was learning that a useful voice agent is a system.
It needs:
Speech recognition.
Reasoning.
Fast speech synthesis.
Memory.
Tools.
Safety boundaries.
Human escalation.
Observability.
And eventually, multiple specialized agents working together.
MoneyBuddy started as a voice that could answer me.
It ended as a small AI customer-service team.
And that was the most exciting part of the entire challenge.
GitHub:
https://github.com/tanush326k/murf-livekit-starter
Challenge:
10 Days of Voice Agents β VoiceForBharat Edition
TTS:
Murf Falcon
If you're building a voice agent too, I'd love to hear what you're working on.
Building this over ten days taught me something I didn't expect:
The hard part of voice AI isn't making an AI speak.
The hard part is making the conversation useful, safe, fast, and predictable.
That's where the real engineering begins.
_