Build an AI Receptionist With Whisper, Claude, Chatterbox and LiveKit: Architecture, Latency Budget and Starter Code

A developer has published a reference architecture for an AI phone receptionist built on LiveKit Agents for telephony and turn-taking, Whisper for speech-to-text, Claude for reasoning and tool calls, and Resemble AI's Chatterbox for streamed text-to-speech. The writeup lays out a per-stage latency budget from end-of-turn detection to first TTS audio, plus starter Python code defining function tools for checking slots, booking appointments and taking messages. It cites Patient Prism data that 31% of callers to 8,280 dental locations hung up before reaching anyone.

This post is AI-assisted. Most "AI receptionist" tutorials stop at a chatbot that talks. A receptionist a dental clinic will actually pay for has to pick up a real phone line, understand messy speech, follow the clinic's rules, book into a real calendar and do all of it fast enough that the caller never notices a gap. This is the architecture we use as a starting point, built from four pieces you can run yourself: | Job | Tool | License | |---|---|---| | Phone line, streaming, turn taking | LiveKit Agents https://github.com/livekit/agents | Apache 2.0 turn detector model under the LiveKit Model License | | Speech to text | Whisper https://github.com/openai/whisper | MIT | | Reasoning and tool calls | Claude Anthropic API | Paid API | | Text to speech, voice cloning | Chatterbox by Resemble AI https://huggingface.co/ResembleAI/chatterbox | MIT | Patient Prism's analysis of 11.5 million calls to 8,280 dental locations found 31% of callers hung up before reaching anyone report https://www.patientprism.com/report/dental-patient-access-report/ . Only 42% of consumers say they leave a voicemail CallRail survey https://www.callrail.com/blog/missed-calls-cost-businesses-more-than-ever . Missed calls are lost bookings, which makes "answer every call" an easy thing to sell. The business math people quote is 100 clinics × $300 a month = $30,000 MRR. The engineering below is what makes that number survivable. PSTN caller │ SIP trunk ▼ LiveKit server room per call │ audio frames ▼ Agent worker Python, LiveKit Agents ├─ VAD + turn detector → decides "caller finished" ├─ STT adapter → faster-whisper on GPU streaming chunks ├─ LLM → Claude with tools: check slots, book, take message, transfer └─ TTS adapter → Chatterbox clinic's consented voice , streamed sentence by sentence │ ▼ Clinic systems: calendar / practice software / CRM / SMS Key design choices: book appointment , your code books it, and the result goes back to the model. Callers start feeling awkward somewhere past a second of silence. Rough budget from "caller stops talking" to "agent audio starts": | Stage | Target | |---|---| | End of turn detection | 200 to 400 ms | | Final transcript chunk | 100 to 300 ms | | Claude first tokens short system prompt, cached rules | 300 to 700 ms | | First sentence of TTS audio | 150 to 400 ms | These are targets to measure against, not guarantees. Measure each stage per call and log it. The biggest wins usually come from shorter prompts, streaming TTS by sentence and keeping the GPU workers warm. LiveKit's starter template lk agent init my-agent --template agent-starter-python gives you a working agent with hosted models. The API evolves quickly, so treat the snippet below as the shape of the code and check the current voice AI quickstart https://docs.livekit.io/agents/start/voice-ai/ for exact signatures. python from livekit import agents from livekit.agents import Agent, AgentServer, AgentSession, function tool from adapters import WhisperSTT, ClaudeLLM, ChatterboxTTS your thin adapters CLINIC RULES = """You are the receptionist for Bright Smile Dental. Hours: Mon to Sat, 9am to 6pm. Book cleanings, checkups and consultations only. Never give medical advice. Transfer emergencies, billing disputes and upset callers to a human. Keep answers under two sentences.""" class Receptionist Agent : def init self : super . init instructions=CLINIC RULES @function tool async def check slots self, date: str - list str : """Return free appointment slots for a date YYYY-MM-DD .""" return await calendar.free slots date @function tool async def book appointment self, name: str, phone: str, slot: str, reason: str - str: """Book a slot and send an SMS confirmation.""" booking = await calendar.book name, phone, slot, reason await sms.send phone, f"Confirmed: {slot} at Bright Smile Dental" return booking.id server = AgentServer @server.rtc session agent name="receptionist" async def entrypoint ctx: agents.JobContext : session = AgentSession stt=WhisperSTT model="large-v3", device="cuda" , llm=ClaudeLLM , tts=ChatterboxTTS voice ref="voices/clinic consented.wav" , await session.start room=ctx.room, agent=Receptionist await session.generate reply instructions="Greet the caller and ask how you can help." if name == " main ": agents.cli.run app server The adapters are small classes that implement LiveKit's STT, LLM and TTS interfaces around faster-whisper, the Anthropic SDK and the Chatterbox Python package. Writing them yourself keeps every model swappable. The models are commodities now. The value is in the boring parts: integrations with each niche's software, reliable transfers, and reporting that shows the owner how many calls were saved. Pick one niche, nail those three, and the $30K MRR math stops being a slide. Axionry is an AI product development studio that builds production voice agents. See the AI receptionist build https://axionry.com/build/ai-receptionist , our voice AI development https://axionry.com/services/voice-ai-development service, or price your setup with the voice AI cost calculator https://axionry.com/tools/voice-ai-cost-calculator at axionry.com https://axionry.com .