{"slug": "i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it", "title": "I Built an AI Voice Sales Agent — Here’s the Architecture Behind It", "summary": "A developer has documented the architecture for an AI voice sales agent that lets dealers place product orders through a phone conversation, combining Twilio telephony, the OpenAI Realtime API, and LangGraph for stateful workflow orchestration. The system routes requests through discrete business tools such as search_product(), check_inventory(), and create_draft_order(), backed by a FastAPI service with PostgreSQL and Redis, before pushing confirmed orders into an ERP. The developer emphasizes that the LLM is not treated as the source of truth, with real-time voice chosen to avoid the multi-second latency of a conventional speech-to-text pipeline.", "body_md": "I Built an AI Voice Sales Agent — Here’s the Architecture Behind It\n\nWhat if a dealer could simply call a number, tell an AI what they need, and place an order through a normal conversation?\n\nNo app.\n\nNo searching through products.\n\nNo filling out forms.\n\nJust talk.\n\nThat was the idea behind a project I've been working on: an AI Voice Sales Agent for dealers.\n\nFor example:\n\n\"I need 50 boxes of Product X and 20 of Product Y.\"\n\nThe AI should understand the request, check whether the products are available, figure out the dealer's pricing, apply any applicable schemes, confirm the order, and eventually push it into the company's ERP.\n\nThis isn't a finished enterprise product yet. I'm still building and figuring things out, but I wanted to document how I'm approaching the architecture and some of the decisions I've made along the way.\n\nThe Architecture\n\nThe basic flow looks something like this:\n\nDealer\n\n   ↓\n\nPhone Call\n\n   ↓\n\nTwilio\n\n   ↓\n\nOpenAI Realtime API\n\n   ↓\n\nLangGraph\n\n   ↓\n\nBusiness Tools\n\n   ↓\n\nFastAPI Backend\n\n   ↓\n\nPostgreSQL / Redis\n\n   ↓\n\nERP\n\nThe interesting part isn't just getting an LLM to talk.\n\nThe AI actually needs to do things.\n\nIt should be able to:\n\nLet's break down how I'm thinking about each part.\n\nThe first challenge is simple:\n\nHow does the dealer actually talk to the system?\n\nI'm using Twilio as the telephony layer.\n\nThe basic flow is:\n\nDealer calls\n\n    ↓\n\nTwilio receives the call\n\n    ↓\n\nAudio goes to the AI system\n\n    ↓\n\nAI processes the conversation\n\n    ↓\n\nResponse is generated\n\n    ↓\n\nDealer hears the response\n\nInitially, I thought of voice as simply:\n\nSpeech → Text → LLM → Text → Speech\n\nBut once you start thinking about an actual conversation, latency becomes a huge deal.\n\nImagine saying something to an AI and waiting 4–5 seconds for every response.\n\nTechnically, it works.\n\nAs a conversation?\n\nNot great.\n\nThat's why I'm looking at real-time voice capabilities rather than treating the system like a normal chatbot with a microphone attached to it.\n\nFor the conversational intelligence, I'm using the OpenAI Realtime API.\n\nBut here's something I realized pretty quickly:\n\nThe LLM shouldn't be responsible for everything.\n\nFor example, if a dealer says:\n\n\"Give me 50 of the blue ones.\"\n\nThe AI needs to understand what \"blue ones\" refers to.\n\nThat requires conversation context.\n\nThe system might have something like:\n\nDealer:\n\nABC Distributors\n\nCurrent conversation:\n\nProduct: Product X\n\nVariant: Blue\n\nRequested quantity: 50\n\nDealer preferences:\n\nPreferred warehouse: Pune\n\nPreferred language: English\n\nSo the AI understands the conversation instead of treating every sentence as an isolated question.\n\nThis is probably one of the parts I'm most interested in.\n\nI don't want the LLM to directly interact with my database and randomly decide what to do.\n\nInstead, I'm giving the agent specific tools.\n\nsearch_product()\n\ncheck_inventory()\n\nget_dealer_price()\n\nget_scheme()\n\nget_customer_history()\n\ncreate_draft_order()\n\nSo a conversation could look something like:\n\nDealer:\n\n\"I need 50 units of Product X.\"\n\nAI:\n\n\"Let me check the availability.\"\n\n```\n    ↓\n```\n\ncheck_inventory(\"Product X\")\n\n```\n    ↓\n```\n\nInventory:\n\n72 units available\n\n```\n    ↓\n```\n\nAI:\n\n\"We have 72 units available. Would you like me to create the order for 50?\"\n\n```\n    ↓\n```\n\nDealer:\n\n\"Yes.\"\n\n```\n    ↓\n```\n\ncreate_draft_order(...)\n\nThis is where LangGraph becomes useful.\n\nInstead of having one massive prompt trying to handle the entire business workflow, the agent can move through different states and use specific tools.\n\nConceptually:\n\nSTART\n\n  ↓\n\nUnderstand Request\n\n  ↓\n\nIdentify Product\n\n  ↓\n\nCheck Inventory\n\n  ↓\n\nCheck Dealer Pricing\n\n  ↓\n\nApply Scheme\n\n  ↓\n\nConfirm Order\n\n  ↓\n\nCreate Draft Order\n\n  ↓\n\nEND\n\nThis also makes debugging much easier.\n\nIf something goes wrong, I can ask:\n\nWhich step failed?\n\nInstead of:\n\nWhy did the AI randomly do that?\n\nOne thing I definitely don't want is for the AI to become the source of truth.\n\nThe LLM can understand things.\n\nIt can reason.\n\nIt can communicate.\n\nBut it shouldn't be the database.\n\nThe system needs structured data for things like:\n\nDealer\n\n ├── Contacts\n\n ├── Addresses\n\n ├── Credit Limit\n\n ├── Language Preference\n\n └── Preferred Warehouse\n\nProduct\n\n ├── Product Variant\n\n ├── SKU\n\n ├── Pricing\n\n └── Inventory\n\nI'm using PostgreSQL for this persistent data.\n\nI'm also using UUIDs for primary keys and keeping things like timestamps, foreign keys, indexes, and soft-delete support in the domain design.\n\nI'm still refining the schema, but getting this foundation right is important because adding AI on top of a messy data model isn't going to magically fix it.\n\nNot everything needs to be stored permanently in PostgreSQL.\n\nA voice conversation can have a lot of temporary state.\n\nThat's where Redis comes in.\n\nThe mental model I'm using is basically:\n\nPostgreSQL\n\n= Persistent business data\n\nRedis\n\n= Fast temporary state + caching\n\nLet's say a dealer says:\n\n\"I need 100 units.\"\n\nThe AI can't just say:\n\n\"Sure, I've placed the order.\"\n\nIt needs to check what's actually available.\n\nSomething like:\n\nDealer Request\n\n      ↓\n\nIdentify SKU\n\n      ↓\n\nInventory Service\n\n      ↓\n\nAvailable Quantity\n\n      ↓\n\nAI Response\n\nIf the system only has 60 units, the AI should say:\n\n\"We currently have 60 units available. Would you like me to create the order for 60?\"\n\nThis is a principle I'm trying to stick to throughout the project:\n\nThe AI should reason about business data, not invent business data.\n\nThis is another place where a normal chatbot approach isn't enough.\n\nIn B2B, everyone doesn't necessarily get the same price.\n\nDifferent dealers might have:\n\nSo the flow could look like:\n\nBase Product Price\n\n        ↓\n\nDealer-specific Price\n\n        ↓\n\nApplicable Scheme\n\n        ↓\n\nDiscount\n\n        ↓\n\nFinal Price\n\nBut here's an important architectural decision:\n\nThe LLM shouldn't calculate the final business-critical price itself.\n\nThe backend should do that.\n\nThe AI can explain the result to the dealer.\n\nThe actual calculation should come from deterministic business logic.\n\nThis is probably one of the coolest parts of the idea.\n\nA useful sales agent shouldn't feel like it has amnesia after every phone call.\n\nSuppose a dealer usually orders a particular product or prefers a particular warehouse.\n\nThe system could remember useful information such as:\n\nThen a future conversation could be much smoother.\n\nInstead of asking:\n\n\"Which warehouse do you want?\"\n\nevery single time, the system could already know the dealer's preferred warehouse and simply confirm it when needed.\n\nBut there's an important distinction here.\n\nNot everything the dealer says should automatically become permanent memory.\n\nMemory needs rules.\n\nSome information is temporary conversation context.\n\nSome information is actual customer data.\n\nAnd some information probably shouldn't be stored at all.\n\nThat's something I want to handle carefully as the project develops.\n\nEventually, the AI needs to connect with the systems that the business already uses.\n\nThat's where ERP integration comes in.\n\nThe architecture I'm aiming for is:\n\nAI Agent\n\n    ↓\n\nBackend\n\n    ↓\n\nBusiness Logic\n\n    ↓\n\nERP APIs\n\n    ↓\n\nOrders / Inventory / Customers\n\nI don't want the AI agent directly modifying ERP data.\n\nInstead:\n\nAI\n\n ↓\n\nTool\n\n ↓\n\nBackend Validation\n\n ↓\n\nERP\n\nThat gives us a much safer boundary between the unpredictable nature of AI and the deterministic nature of enterprise systems.\n\nThis is probably one of the biggest things I've learned while designing this.\n\nIt can be tempting to just give an LLM database access and say:\n\n\"Do whatever you need.\"\n\nBut for a system dealing with real orders, pricing and customer information, that's a very bad idea.\n\nI'd rather have:\n\n❌ LLM → Database\n\n✅ LLM → Tool → Backend → Database\n\nLLM\n\n ↓\n\nget_inventory(\"SKU123\")\n\n ↓\n\nBackend validates request\n\n ↓\n\nDatabase query\n\n ↓\n\nStructured result\n\n ↓\n\nLLM\n\nNow I have a proper place for:\n\nAnd most importantly, I know exactly what the AI is allowed to do.\n\nThe Tech Stack\n\nFrontend: Next.js\n\nStyling: Tailwind CSS + shadcn/ui\n\nBackend: FastAPI\n\nDatabase: PostgreSQL\n\nCache / State: Redis\n\nORM: SQLAlchemy\n\nMigrations: Alembic\n\nVoice: Twilio\n\nReal-time AI: OpenAI Realtime API\n\nAgent Orchestration: LangGraph\n\nVector Search: pgvector\n\nVersion Control: Git + GitHub\n\nMonitoring: Sentry + OpenTelemetry\n\nI'm trying to avoid choosing technologies just because they're popular.\n\nI want every piece of the stack to have a reason for being there.\n\nWhat I'm Still Figuring Out\n\nThe project is still a work in progress, so there are a lot of things I'm actively figuring out.\n\nA voice agent has to feel like a conversation.\n\nEven a technically correct answer feels bad if it takes too long.\n\nThe agent absolutely cannot randomly invent:\n\nThose things need to come from actual systems.\n\nWhat happens when the dealer says:\n\n\"No, not that one. The other blue one.\"\n\nThe agent needs enough context to understand what they're referring to.\n\nWhat if the inventory service is down?\n\nThe AI shouldn't pretend that everything worked.\n\nIt needs to understand that the tool failed and communicate that properly.\n\nOnce you're dealing with actual business transactions, security becomes a major part of the architecture.\n\nThings like:\n\ncan't just be an afterthought.\n\nThe Biggest Thing I've Learned\n\nThe biggest lesson from this project so far is:\n\nBuilding an AI application isn't the same as putting an LLM inside an application.\n\nThe LLM is only one part of the system.\n\nA useful AI product needs:\n\nAI\n\n+\n\nBusiness Logic\n\n+\n\nData\n\n+\n\nTools\n\n+\n\nState\n\n+\n\nSecurity\n\n+\n\nObservability\n\n+\n\nReliable Infrastructure\n\nThe model provides the intelligence.\n\nBut the architecture around it provides the reliability.\n\nAnd I think that's an important distinction, especially when moving from AI demos to actual products.\n\nWhat's Next?\n\nRight now, I'm focusing on building the foundation properly before trying to make everything \"smart.\"\n\nThe next things on my list are:\n\nThe end goal is pretty simple:\n\nA dealer should be able to pick up a phone and complete a business transaction through a natural conversation.\n\nCall\n\n ↓\n\nTalk\n\n ↓\n\nConfirm\n\n ↓\n\nOrder\n\nThat's the experience I'm trying to build.\n\nFinal Thoughts\n\nThis project has also changed the way I think about software engineering.\n\nEarlier, I mostly thought about applications as:\n\nFrontend\n\n+\n\nBackend\n\n+\n\nDatabase\n\nNow I'm asking a lot more questions:\n\nWhat should the AI be allowed to decide?\n\nWhat should the backend decide?\n\nWhere does the actual source of truth live?\n\nWhat happens when the AI is wrong?\n\nWhat happens when a tool fails?\n\nHow do we recover from a misunderstood request?\n\nHow do we make the whole thing observable?\n\nAnd honestly, I'm still figuring out many of these answers.\n\nThat's probably my favorite part of building this.\n\nI'm not trying to pretend I have the perfect architecture figured out.\n\nI'm building it, breaking things, learning, and improving it as I go.\n\nBuild → Break → Learn → Improve.\n\nIf you're also building AI agents, voice applications, or AI-powered SaaS products, I'd love to hear what you're working on and what problems you've run into.\n\nLet's learn from each other. 🚀", "url": "https://wpnews.pro/news/i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it", "canonical_source": "https://dev.to/agrima-06/i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it-g4g", "published_at": "2026-09-15 08:55:51+00:00", "updated_at": "2026-09-15 09:09:13.451137+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "large-language-models", "natural-language-processing", "developer-tools"], "entities": ["Twilio", "OpenAI Realtime API", "LangGraph", "FastAPI", "PostgreSQL", "Redis"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it", "markdown": "https://wpnews.pro/news/i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it.md", "text": "https://wpnews.pro/news/i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it.txt", "jsonld": "https://wpnews.pro/news/i-built-an-ai-voice-sales-agent-heres-the-architecture-behind-it.jsonld"}}