How do production voice agents (e.g. Bland.ai) orchestrate prompts and conversation flow without relying on huge hand-written system prompts?
I’m building a real-time voice agent, and I’m trying to understand how production platforms like Bland.ai, Retell, Vapi, etc. handle prompt orchestration internally.
In my current implementation, I have a base system prompt that is cached, and I perform state-wise prompt injection depending on where the user is in the conversation (e.g., booking, checking availability, Q&A, BANT qualification, etc.).
The problem I’m facing is that I still end up writing a lot of explicit instructions for edge cases. For example:
- If the user asks a question while I’m collecting their email, answer the question and then continue collecting the email.
- If the user changes the topic, handle it and return to the previous task.
- If the user provides partial information, ask only for the missing fields.
- If the user interrupts, resume the previous workflow naturally.
As more edge cases appear, the system prompt keeps growing. Even after carefully covering many scenarios, real users still behave in unexpected ways. Sometimes the model forgets a step, skips required information, or drifts away from the intended workflow.
On the other hand, platforms like Bland.ai expose a simple dashboard where you specify things like:
- Agent name
- Voice/Gender
- Purpose
- Guardrails
- Business instructions
…and the agent seems to work surprisingly well without the developer manually encoding hundreds of conversational edge cases.
This makes me wonder:
Questions
- Are these platforms actually generating a much larger prompt behind the scenes?
- Are they using hierarchical prompts or prompt compilation?
- Is there an orchestration engine that dynamically injects instructions based on the conversation state?
- Are finite-state machines (FSMs), behavior trees, workflow graphs, or planners commonly used?
- How do they prevent the LLM from drifting while still allowing natural conversation?
- How do they recover after interruptions or topic changes without stuffing every possible scenario into the system prompt?
- Do they rely heavily on tool calling and external orchestration instead of prompt engineering?
- How are prompts compiled or transformed from the dashboard configuration into something that reliably handles real-world conversations?
One thing I’ve noticed is that if I simplify my prompt to something like:
“Answer the user’s questions and continue with the booking.”
the model often becomes too conversational and eventually stops driving the booking process. So there seems to be a balance between giving the model autonomy and enforcing deterministic business logic.
My Current Architecture
Backend: FastAPI STT: Sarvam Speech-to-Text LLM: Streaming TTS: Sarvam Text-to-Speech Turn Detection: Pipecat SmartTurn V3 Voice Activity Detection: Silero VAD
Streaming Pipeline
User Speech
│
▼
Sarvam STT
│
▼
LLM
│
▼
Sarvam TTS
│
▼
User
What I’m Trying to Achieve
My goal is to build an orchestration layer that:
- Maintains deterministic workflows while still allowing natural conversations.
- Can gracefully handle interruptions, off-topic questions, and unexpected user behavior.
- Doesn’t require an ever-growing monolithic system prompt.
- Achieves sub-second latency suitable for production voice agents.
I’m less interested in vendor-specific implementations and more interested in the architectural patterns used in production systems.
For people who have built production-grade conversational or voice AI systems:
- How do you structure the orchestration layer?
- How much of the logic lives in prompts versus external code?
- What patterns have worked well for keeping the conversation on track while remaining flexible?
- If you were building a voice agent from scratch today, how would you design this layer?
Hey @John6666 can you throw some light in this, I have read from countless forums and docs and still they do not seem to give a concrete explanation, the issue is with the orchestration layer and not the components