How I Built a Bilingual RAG-Powered AI Calling & Chat Agent (With a Full Admin Ops Center)

A developer built a bilingual, multi-channel AI calling and chat agent that routes phone, web chat, WhatsApp, and Instagram traffic through a single RAG pipeline backed by ChromaDB and gemini-embedding-001. The system uses a confidence threshold to answer from the knowledge base or fall back to live web search, tagging each interaction with its source to produce a real-time KB Hit Rate metric. It also includes a no-code admin operations center with a diagnostic playground, aggregate dashboards, conversational lead extraction, and runtime business-variable branding for multi-tenant deployments.

Most "AI chatbot" projects stop at a demo: a widget, an OpenAI/Gemini call, a canned prompt. Getting one into production for real businesses — across voice calls, web chat, and WhatsApp, in two languages, without hallucinating — is a different problem. Here's how I approached it. The Core Challenge Three requirements shaped the whole architecture: Multi-channel, single brain — phone calls, web chat, WhatsApp, and Instagram all needed to hit the same knowledge base and produce consistent answers. No hallucination tolerance — a wrong answer on a live sales call is worse than no answer. Non-technical operators — the business owner using the admin panel should never need to touch a config file, an API key in code, or a terminal. The RAG Pipeline The knowledge layer runs on a fairly standard but carefully tuned RAG stack: Ingestion: a website crawler pulls headers, paragraphs, and lists from a given URL or a pasted/uploaded document , and normalizes it into clean text chunks. Embedding: chunks are embedded using gemini-embedding-001. Storage: vectors are indexed into ChromaDB for fast cosine-similarity retrieval. Retrieval-time logic: on every incoming query, the system computes similarity scores against the knowledge base. If the top match clears a confidence threshold, it answers from the KB source: knowledge base . If not, it falls back to a live, relevant web search source: search fallback rather than letting the LLM freestyle an answer from parametric memory. That fallback tagging turned out to be one of the most useful design decisions — every logged interaction carries a source field knowledge base or search fallback , which means you get a real-time KB Hit Rate metric for free: how often the system is confidently answering from your own data vs. reaching outside it. Observability Was Not an Afterthought A lot of RAG demos skip this, and it's the first thing that breaks trust once you hand a system to a real business. I built a diagnostic layer that surfaces, per query: Cosine distance / similarity scores against retrieved chunks Which knowledge base sections were actually matched End-to-end latency ms Source attribution KB vs. fallback This is exposed directly in a "Diagnostic Playground" in the admin UI — type a query, and see exactly what the retrieval layer matched and why, before it ever reaches a real customer. Aggregate metrics Total Conversations, Average Latency, KB Hit Rate, Search Fallback %, Error Rate roll up from the same interaction logs into an overview dashboard — no separate analytics pipeline needed. Lead Extraction Without a Structured Form Because conversations happen in free text and voice-to-text , lead capture couldn't rely on form fields. The system parses conversational turns for identifiers — name, phone number, email — as they're mentioned naturally "my name is X and my number is Y" , writes them to a leads table, deduplicates against existing entries, and assigns the lead to a sales rep via round-robin rotation. This runs as a lightweight side-effect of the main conversation loop, not a separate workflow the user has to trigger. Multi-Tenant Branding Without Multi-Tenant Infra Complexity Rather than spinning up separate deployments per client, business identity company name, agent persona, tone, contact details, tagline is stored as a set of "Business Variables" that get interpolated into the system prompt and voice/chat responses at runtime. One codebase, many brands — which matters a lot if you're an agency or planning to white-label this. No-Code Data Layer The default store is local SQLite for simplicity, but the admin UI also supports connecting an external Postgres-compatible database Supabase, Neon, or vanilla Postgres via a connection string pasted directly into the UI — no backend redeploy required. This was a deliberate trade-off: less "clever" than an ORM migration system, but it means a non-engineer can point the whole system at their own cloud database in under a minute. Since the target users span Hindi and English speakers, language handling isn't a toggle — the model detects and responds in whichever language the user used, per message, in both voice STT/TTS and text channels. What I'd Do Differently Next Real-time in-call escalation to a human when confidence is low currently: unresolved queries become a lead for post-call follow-up — live handoff is the next milestone Per-channel confidence thresholds a WhatsApp typo tolerance vs. a live voice transcript need different tuning Try It If you're building something similar or want to see this running on a real business's data, I opened up a free 10-day trial — happy to walk through the architecture in more depth too. 📧 nexopersupport@gmail.com mailto:nexopersupport@gmail.com · 🌐 nexoper.in Would genuinely love feedback from anyone who's tuned RAG confidence thresholds for production voice use cases — what's worked for you?