Building a Hinglish voice mentor with Gemini and LiveKit is a

A developer built a Hinglish voice mentor using Google Gemini 1.5 Flash-Lite, Deepgram nova-3, Murf Falcon, and LiveKit Agents, enabling real-time spoken tutoring on topics like backpropagation and RAG. The system features persistent memory, specialist handoff, outbound calls, and human escalation, with a Next.js frontend and canvas wave-visualizer to manage the 'silence gap'.

Building a Hinglish voice mentor with Gemini and LiveKit is a The goal here wasn't just a chatbot with a voice skin, but a full LLM agent capable of teaching complex topics like backpropagation, RAG /en/tags/rag/ , and vector embeddings through natural, spoken conversation. The Technical Stack To keep the latency low enough that the conversation doesn't feel like a walkie-talkie from 1995, the system uses a specific pipeline: STT: Deepgram nova-3 . This is critical because it handles code-switching mixing Hindi and English in real-time without needing a manual language toggle. LLM: Google Gemini /en/tags/gemini/ 1.5 Flash-Lite. In voice workflows, speed beats raw reasoning power. If the model takes 3 seconds to think, the human has already checked out. TTS: Murf Falcon. Orchestration: LiveKit Agents for the real-time transport layer. Feature Set and Agent Logic What makes this more than a basic wrapper is the implementation of agentic workflows. It's not just a linear prompt; it's a system with state and tools. Persistent Memory: It tracks learner progress and common mistakes, meaning it doesn't ask you to explain what a vector is three times in one session. Specialist Handoff: This is a classic LLM agent pattern. When the general mentor hits a complexity ceiling, it transfers the session to a "RAG Deep-Dive Specialist" or an "Interview Prep Specialist." Outbound Capabilities: The system can initiate calls for scheduled practice, moving the AI from a passive tool to an active coach. Human Escalation: A built-in trigger that creates a mentor request when the LLM detects the user is genuinely stuck. Practical Deployment Architecture If you're trying to build a similar real-world AI workflow, the data flow looks like this: User Audio → Deepgram STT → Gemini LLM + Tool Use → Murf TTS → User Audio ↓ External DB / Practice Exercises For anyone attempting a deployment from scratch, the biggest hurdle isn't the LLM—it's the "silence gap." To make it feel human, you need an STT that handles interruptions gracefully and an LLM that can generate concise responses. Long-winded AI paragraphs are death for voice agents. The project also utilizes a Next.js frontend with a custom HTML5 canvas wave-visualizer. While it seems like eye candy, it's actually a functional necessity; in voice AI, the user needs a visual cue that the agent is "listening" or "thinking" to prevent them from talking over the bot. Example of the core logic flow for the agent 1. Listen for voice activity VAD 2. Transcribe via Deepgram nova-3 3. Process intent with Gemini 1.5 Flash 4. If specialized knowledge is needed - Handoff to Specialist Agent 5. Convert text response to speech via Murf Next Claude's invisible watermarking doesn't actually prove a human → /en/threads/6465/