I expected to spend a weekend on working on my vision. Instead, I had a voice agent asking me "How was your day?" in under 15 minutes. What surprised me most wasn't the speed - it was that when I interrupted the voice AI agent I named "Compass" mid-sentence to change my answer, she just stopped and listened. No stuttering, no doubled audio, no ghost speech finishing in the background. It just worked, out of the box, without a single line of interruption-handling code on my end.
If you've tried to build a voice agent from scratch, you know the pipeline looks deceptively simple on a whiteboard: speech in, text to LLM, speech out. The reality is messier.
You need a WebRTC or WebSocket layer to stream audio in real time. You need to integrate STT (and handle partial transcripts), stream tokens to TTS, manage token refresh, handle network retries, detect when the user starts speaking mid-sentence, and somehow prevent the agent from continuing its TTS output while the user is already replying. That's before you write a single line of actual product logic.
The Agora Agents SDK removes every item on that list. It's built on top of Agora's existing RTC infrastructure, which is the same real-time communications network that powers video calling for hundreds of millions of users. The SDK wraps that into a Python (or TypeScript or Go) library where you describe what your agent should do, not how the audio pipeline should work.
Daily Reflection Companion
a voice AI named Compass that guides users through a structured end-of-day reflection: a daily check-in, a meaningful moment exploration, a gratitude round, a forward intention, and then an AI-generated written summary of the conversation.
Install
pip install agora-agents fastapi uvicorn[standard] python-dotenv openai
That's it. The SDK installs in under 10 seconds. Imports are under agora_agent (note: no hyphen at import time).
Credentials You'll Need
Copy your .env.example to .env:
AGORA_APP_ID=your_app_id
AGORA_APP_CERTIFICATE=your_certificate
OPENAI_API_KEY=sk-…
DEEPGRAM_API_KEY=your_deepgram_key
Note: The Agora console's Conversational AI toggle is not prominently labelled. I spent about 4 minutes finding it - it lives under Project Settings → Features. Once enabled, everything worked on the first try.
The builder chain
The entire STT → LLM → TTS pipeline is configured in one fluent chain. Here's the real code from agent.py:
from agora_agent import (
Agent, Agora, Area,
DeepgramSTT, OpenAI as AgoraOpenAI, MiniMaxTTS,
expires_in_hours,
)
client = Agora(area=Area.US, app_id=app_id, app_certificate=app_certificate)
agent = (
Agent(client=client, turn_detection={"language": "en-US"})
.with_stt(DeepgramSTT(model="nova-3", language="en"))
.with_llm(
AgoraOpenAI(
model="gpt-4o-mini",
system_messages=[{"role": "system", "content": REFLECTION_SYSTEM_PROMPT}],
greeting_message="Hello! I'm Compass. How was your day today?",
failure_message="I didn't quite catch that. Could you say that again?",
max_history=50,
params={"max_tokens": 150, "temperature": 0.75},
)
)
.with_tts(MiniMaxTTS(model="speech_2_6_turbo", voice_id="English_captivating_female1"))
)
Every line is intentional:
turn_detection={"language": "en-US"} - tells the VAD (Voice Activity Detection) which language's speech patterns to use for end-of-turn detection. This directly affects how quickly the agent recognises you've finished speaking.max_history=50 - the agent's LLM receives up to 50 turns of conversation as context. Critical for a reflection agent that needs to remember what was said early in the session.max_tokens=150 - voice replies need to be short. Capping at 150 tokens enforces this at the model level, not just the prompt.greeting_message - the agent speaks this immediately when the session starts, without waiting for the user to speak first. No extra session.say() call needed.
The reflection flow is driven entirely by the LLM system prompt. No explicit state machine, no conditional logic in the server code. The prompt instructs the agent to move through five phases naturally:
PHASE 1 - DAILY CHECK-IN: Ask one meaningful follow-up after the user's response.
PHASE 2 - MEANINGFUL MOMENT: Explore one significant experience.
PHASE 3 - GRATITUDE: Ask for 2–3 things they're thankful for.
PHASE 4 - TOMORROW'S INTENTION: Ask for one thing they'd like to carry forward.
PHASE 5 - CLOSING: Offer a warm, personalised goodbye.
VOICE RULES: Keep all responses under 35 words. One question per turn. No markdown.
The "under 35 words" constraint was the most important prompt engineering decision. Voice synthesis doesn't render markdown, and long replies feel like lectures, not conversations.
Starting a Session
session = agent.create_session(
channel=f"reflection-{int(time.time())}",
agent_uid="999",
remote_uids=["*"],
idle_timeout=90,
expires_in=expires_in_hours(1),
)
agent_id = session.start()
session.start() is a single blocking call that provisions the agent, connects it to the RTC channel, and returns an agent_id. The channel name is how the browser's Agora Web SDK joins the same audio room.
After the conversation ends, I call session.get_history() to retrieve the transcript and send it to GPT-4o-mini with a structured summary prompt:
history = session.get_history()
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": SUMMARY_PROMPT},
{"role": "user", "content": f"Transcript:\n\n{transcript}"},
],
max_tokens=200,
)
return response.choices[0].message.content
The summary appears on-screen after the session ends - a written record of what you reflected on. This is the feature that makes this more than a demo.
The browser uses the Agora Web SDK (loaded from CDN - no npm, no webpack) to join the same RTC channel:
// Create client and join
rtcClient = AgoraRTC.createClient({ mode: "rtc", codec: "vp8" });
await rtcClient.join(app_id, channel, token || null, uid);
// Publish microphone
localMicTrack = await AgoraRTC.createMicrophoneAudioTrack({
encoderConfig: "speech_standard",
});
await rtcClient.publish(localMicTrack);
// Subscribe to agent's audio and play it
rtcClient.on("user-published", async (user, mediaType) => {
if (mediaType !== "audio") return;
await rtcClient.subscribe(user, "audio");
user.audioTrack.play();
});
Three async calls: join, publish, subscribe. That's the entire WebRTC layer.
Once the pipeline runs, the question that matters is: does it feel like a real conversation?
Turn-taking
The Deepgram nova-3 model detects end-of-speech accurately. Compass waits for me to finish, then responds within about 1.2–1.8 seconds (STT transcription + LLM generation + TTS synthesis). Agora's published benchmark is <650ms end-to-end - I measured closer to 1.2–1.5s for GPT-4o-mini with MiniMax TTS, which still feels conversational rather than laggy.
Interruption test
This is where I was genuinely surprised. I spoke over Compass mid-sentence. She stopped. Immediately, cleanly, no audio bleed. I tried this five times with different timing - sometimes right as she started a word, sometimes deep into a sentence. Every time the cut was instant.
This is handled entirely by Agora's RTC infrastructure and the VAD layer. I wrote zero interruption-handling code. It just worked.
Conversation quality
The reflection flow felt natural across few turns. The agent stayed on-topic, transitioned between phases at the right moments, and the "under 35 words" constraint kept responses appropriately brief for voice. The one area that needed tuning: Compass occasionally asked two questions in one turn early in testing (prompt refinement fixed this with a firm "Ask only ONE question per turn" rule).
Built with Agora Agents SDK (Python) + OpenAI GPT-4o-mini + Deepgram STT + MiniMax TTS
What's genuinely great
The builder pattern is the right abstraction. Instead of wiring five services together yourself, you declare your pipeline and the SDK handles transport, buffering, retries, and token refresh.
Interruption handling working out of the box is not a small thing - in a DIY build, that's easily a full day of work.
session.think() (not demonstrated in this project's MVP but available) is a powerful primitive: you can inject mid-session instructions into the LLM without the agent speaking them aloud. For a more sophisticated reflection agent, this could be used to nudge the agent toward a specific phase based on time elapsed.
What could be better
**The gap between quickstart and production. **The CLI quickstart (agora init) scaffolds a working app fast, but the gap from that template to understanding why each piece exists is steep. Better intermediate documentation (not just API reference, not just quickstart) would help.
**get_history() response format. **The method exists and works, but the response shape isn't clearly documented. I had to handle three possible formats defensively. This is the kind of thing that adds 30 minutes to an otherwise 5-minute task.
Error specificity. When I accidentally misconfigured my App Certificate, the error was a generic 401. A message like "App Certificate mismatch - check AGORA_APP_CERTIFICATE in your environment" would have saved 10 minutes of debugging.
These are fixable problems, not fundamental ones. The core pipeline is rock-solid.
I started this build expecting to spend most of my time on infrastructure. Instead, I spent most of it on the part that matters - the conversation design, the system prompt, the reflection flow. That's the correct trade-off, and the SDK made it possible.
The Agora Agents SDK doesn't replace the REST API - it's built on top of it, and REST stays fully supported. What it does is remove the RTC plumbing so you can focus on what your agent should say and feel like, not on how audio bytes travel between browser and model.
For a real-time voice product where interruption, latency, and audio quality matter, the infrastructure Agora provides is serious. For a developer who wants to build that product in an afternoon rather than a week, this SDK is the fastest path I've found.
pip install agora-agents fastapi uvicorn[standard] python-dotenv openai
cp .env.example .env
uvicorn main:app - reload
GitHub (Python SDK): https://github.com/AgoraIO/agora-agents-python
#VoiceAI #AIagents #Agora #ConversationalAI #ConvoAI #OpenAI #Agora #TTS #STT