# Build an AI Receptionist With Whisper, Claude, Chatterbox and LiveKit: Architecture, Latency Budget and Starter Code

> Source: <https://dev.to/neeraj_sharma_757fbb49aa7/build-an-ai-receptionist-with-whisper-claude-chatterbox-and-livekit-architecture-latency-budget-5eka>
> Published: 2026-10-29 14:06:44+00:00

*This post is AI-assisted.*

Most "AI receptionist" tutorials stop at a chatbot that talks. A receptionist a dental clinic will actually pay for has to pick up a real phone line, understand messy speech, follow the clinic's rules, book into a real calendar and do all of it fast enough that the caller never notices a gap.

This is the architecture we use as a starting point, built from four pieces you can run yourself:

| Job | Tool | License | 
|---|---|---|
| Phone line, streaming, turn taking | [LiveKit Agents](https://github.com/livekit/agents) | Apache 2.0 (turn detector model under the LiveKit Model License) | 
| Speech to text | [Whisper](https://github.com/openai/whisper) | MIT | 
| Reasoning and tool calls | Claude (Anthropic API) | Paid API | 
| Text to speech, voice cloning | [Chatterbox by Resemble AI](https://huggingface.co/ResembleAI/chatterbox) | MIT | 

Patient Prism's analysis of 11.5 million calls to 8,280 dental locations found 31% of callers hung up before reaching anyone ([report](https://www.patientprism.com/report/dental-patient-access-report/)). Only 42% of consumers say they leave a voicemail ([CallRail survey](https://www.callrail.com/blog/missed-calls-cost-businesses-more-than-ever)). Missed calls are lost bookings, which makes "answer every call" an easy thing to sell.

The business math people quote is 100 clinics × $300 a month = $30,000 MRR. The engineering below is what makes that number survivable.

```
PSTN caller
   │  SIP trunk
   ▼
LiveKit server (room per call)
   │  audio frames
   ▼
Agent worker (Python, LiveKit Agents)
   ├─ VAD + turn detector  → decides "caller finished"
   ├─ STT adapter          → faster-whisper on GPU (streaming chunks)
   ├─ LLM                  → Claude with tools: check_slots, book, take_message, transfer
   └─ TTS adapter          → Chatterbox (clinic's consented voice), streamed sentence by sentence
   │
   ▼
Clinic systems: calendar / practice software / CRM / SMS
```

Key design choices:

`book_appointment`, your code books it, and the result goes back to the model.
Callers start feeling awkward somewhere past a second of silence. Rough budget from "caller stops talking" to "agent audio starts":

| Stage | Target | 
|---|---|
| End of turn detection | 200 to 400 ms | 
| Final transcript chunk | 100 to 300 ms | 
| Claude first tokens (short system prompt, cached rules) | 300 to 700 ms | 
| First sentence of TTS audio | 150 to 400 ms | 

These are targets to measure against, not guarantees. Measure each stage per call and log it. The biggest wins usually come from shorter prompts, streaming TTS by sentence and keeping the GPU workers warm.

LiveKit's starter template (`lk agent init my-agent --template agent-starter-python`) gives you a working agent with hosted models. The API evolves quickly, so treat the snippet below as the shape of the code and check the current [voice AI quickstart](https://docs.livekit.io/agents/start/voice-ai/) for exact signatures.

``` python
from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, function_tool

from adapters import WhisperSTT, ClaudeLLM, ChatterboxTTS  # your thin adapters

CLINIC_RULES = """You are the receptionist for Bright Smile Dental.
Hours: Mon to Sat, 9am to 6pm. Book cleanings, checkups and consultations only.
Never give medical advice. Transfer emergencies, billing disputes and upset callers to a human.
Keep answers under two sentences."""

class Receptionist(Agent):
    def __init__(self):
        super().__init__(instructions=CLINIC_RULES)

    @function_tool
    async def check_slots(self, date: str) -> list[str]:
        """Return free appointment slots for a date (YYYY-MM-DD)."""
        return await calendar.free_slots(date)

    @function_tool
    async def book_appointment(self, name: str, phone: str, slot: str, reason: str) -> str:
        """Book a slot and send an SMS confirmation."""
        booking = await calendar.book(name, phone, slot, reason)
        await sms.send(phone, f"Confirmed: {slot} at Bright Smile Dental")
        return booking.id

server = AgentServer()

@server.rtc_session(agent_name="receptionist")
async def entrypoint(ctx: agents.JobContext):
    session = AgentSession(
        stt=WhisperSTT(model="large-v3", device="cuda"),
        llm=ClaudeLLM(),
        tts=ChatterboxTTS(voice_ref="voices/clinic_consented.wav"),
    )
    await session.start(room=ctx.room, agent=Receptionist())
    await session.generate_reply(instructions="Greet the caller and ask how you can help.")

if __name__ == "__main__":
    agents.cli.run_app(server)
```

The adapters are small classes that implement LiveKit's STT, LLM and TTS interfaces around faster-whisper, the Anthropic SDK and the Chatterbox Python package. Writing them yourself keeps every model swappable.

The models are commodities now. The value is in the boring parts: integrations with each niche's software, reliable transfers, and reporting that shows the owner how many calls were saved. Pick one niche, nail those three, and the $30K MRR math stops being a slide.

*Axionry is an AI product development studio that builds production voice agents. See the [AI receptionist build](https://axionry.com/build/ai-receptionist), our [voice AI development](https://axionry.com/services/voice-ai-development) service, or price your setup with the [voice AI cost calculator](https://axionry.com/tools/voice-ai-cost-calculator) at [axionry.com](https://axionry.com).*