{"slug": "voice-ai-agent-construction", "title": "Voice AI agent construction", "summary": "A developer building a real-time voice AI agent struggles with prompt orchestration as edge cases cause system prompts to grow uncontrollably. They question how production platforms like Bland.ai handle conversation flow without massive prompts, suspecting behind-the-scenes prompt compilation, hierarchical prompts, or external orchestration engines. The developer seeks architectural patterns for maintaining deterministic workflows while allowing natural conversation and handling interruptions.", "body_md": "How do production voice agents (e.g. Bland.ai) orchestrate prompts and conversation flow without relying on huge hand-written system prompts?\n\nI’m building a real-time voice agent, and I’m trying to understand how production platforms like Bland.ai, Retell, Vapi, etc. handle prompt orchestration internally.\n\nIn my current implementation, I have a **base system prompt** that is cached, and I perform **state-wise prompt injection** depending on where the user is in the conversation (e.g., booking, checking availability, Q&A, BANT qualification, etc.).\n\nThe problem I’m facing is that I still end up writing a lot of explicit instructions for edge cases. For example:\n\n- If the user asks a question while I’m collecting their email, answer the question and then continue collecting the email.\n- If the user changes the topic, handle it and return to the previous task.\n- If the user provides partial information, ask only for the missing fields.\n- If the user interrupts, resume the previous workflow naturally.\n\nAs more edge cases appear, the system prompt keeps growing. Even after carefully covering many scenarios, real users still behave in unexpected ways. Sometimes the model forgets a step, skips required information, or drifts away from the intended workflow.\n\nOn the other hand, platforms like Bland.ai expose a simple dashboard where you specify things like:\n\n- Agent name\n- Voice/Gender\n- Purpose\n- Guardrails\n- Business instructions\n\n…and the agent seems to work surprisingly well without the developer manually encoding hundreds of conversational edge cases.\n\nThis makes me wonder:\n\nQuestions\n\n- Are these platforms actually generating a much larger prompt behind the scenes?\n- Are they using hierarchical prompts or prompt compilation?\n- Is there an orchestration engine that dynamically injects instructions based on the conversation state?\n- Are finite-state machines (FSMs), behavior trees, workflow graphs, or planners commonly used?\n- How do they prevent the LLM from drifting while still allowing natural conversation?\n- How do they recover after interruptions or topic changes without stuffing every possible scenario into the system prompt?\n- Do they rely heavily on tool calling and external orchestration instead of prompt engineering?\n- How are prompts compiled or transformed from the dashboard configuration into something that reliably handles real-world conversations?\n\nOne thing I’ve noticed is that if I simplify my prompt to something like:\n\n“Answer the user’s questions and continue with the booking.”\n\nthe model often becomes too conversational and eventually stops driving the booking process. So there seems to be a balance between giving the model autonomy and enforcing deterministic business logic.\n\nMy Current Architecture\n\n**Backend:** FastAPI\n**STT:** Sarvam Speech-to-Text\n**LLM:** Streaming\n**TTS:** Sarvam Text-to-Speech\n**Turn Detection:** Pipecat SmartTurn V3\n**Voice Activity Detection:** Silero VAD\n\nStreaming Pipeline\n\n```\nUser Speech\n     │\n     ▼\nSarvam STT\n     │\n     ▼\nLLM\n     │\n     ▼\nSarvam TTS\n     │\n     ▼\nUser\n```\n\nWhat I’m Trying to Achieve\n\nMy goal is to build an orchestration layer that:\n\n- Maintains deterministic workflows while still allowing natural conversations.\n- Can gracefully handle interruptions, off-topic questions, and unexpected user behavior.\n- Doesn’t require an ever-growing monolithic system prompt.\n- Achieves sub-second latency suitable for production voice agents.\n\nI’m less interested in vendor-specific implementations and more interested in the architectural patterns used in production systems.\n\nFor people who have built production-grade conversational or voice AI systems:\n\n- How do you structure the orchestration layer?\n- How much of the logic lives in prompts versus external code?\n- What patterns have worked well for keeping the conversation on track while remaining flexible?\n- If you were building a voice agent from scratch today, how would you design this layer?\n\nHey [@John6666](/u/john6666) can you throw some light in this, I have read from countless forums and docs and still they do not seem to give a concrete explanation, the issue is with the orchestration layer and not the components", "url": "https://wpnews.pro/news/voice-ai-agent-construction", "canonical_source": "https://discuss.huggingface.co/t/voice-ai-agent-construction/177750#post_2", "published_at": "2026-07-13 07:16:44+00:00", "updated_at": "2026-07-13 07:18:35.669705+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-products", "developer-tools", "natural-language-processing"], "entities": ["Bland.ai", "Retell", "Vapi", "Sarvam", "FastAPI", "Pipecat", "Silero"], "alternates": {"html": "https://wpnews.pro/news/voice-ai-agent-construction", "markdown": "https://wpnews.pro/news/voice-ai-agent-construction.md", "text": "https://wpnews.pro/news/voice-ai-agent-construction.txt", "jsonld": "https://wpnews.pro/news/voice-ai-agent-construction.jsonld"}}