The goal here wasn't just a chatbot with a voice skin, but a full LLM agent capable of teaching complex topics like backpropagation, RAG, and vector embeddings through natural, spoken conversation.
The Technical Stack #
To keep the latency low enough that the conversation doesn't feel like a walkie-talkie from 1995, the system uses a specific pipeline:
STT: Deepgramnova-3
. This is critical because it handles code-switching (mixing Hindi and English) in real-time without needing a manual language toggle.LLM: GoogleGemini1.5 Flash-Lite. In voice workflows, speed beats raw reasoning power. If the model takes 3 seconds to think, the human has already checked out.TTS: Murf Falcon.Orchestration: LiveKit Agents for the real-time transport layer.
Feature Set and Agent Logic #
What makes this more than a basic wrapper is the implementation of agentic workflows. It's not just a linear prompt; it's a system with state and tools.
Persistent Memory: It tracks learner progress and common mistakes, meaning it doesn't ask you to explain what a vector is three times in one session.Specialist Handoff: This is a classic LLM agent pattern. When the general mentor hits a complexity ceiling, it transfers the session to a "RAG Deep-Dive Specialist" or an "Interview Prep Specialist."Outbound Capabilities: The system can initiate calls for scheduled practice, moving the AI from a passive tool to an active coach.Human Escalation: A built-in trigger that creates a mentor request when the LLM detects the user is genuinely stuck.
Practical Deployment Architecture #
If you're trying to build a similar real-world AI workflow, the data flow looks like this:
User Audio → Deepgram (STT) → Gemini (LLM + Tool Use) → Murf (TTS) → User Audio
↓
External DB / Practice Exercises
For anyone attempting a deployment from scratch, the biggest hurdle isn't the LLM—it's the "silence gap." To make it feel human, you need an STT that handles interruptions gracefully and an LLM that can generate concise responses. Long-winded AI paragraphs are death for voice agents.
The project also utilizes a Next.js frontend with a custom HTML5 canvas wave-visualizer. While it seems like eye candy, it's actually a functional necessity; in voice AI, the user needs a visual cue that the agent is "listening" or "thinking" to prevent them from talking over the bot.
Next Claude's invisible watermarking doesn't actually prove a human →