I Built an AI Voice Sales Agent — Here’s the Architecture Behind It

A developer has documented the architecture for an AI voice sales agent that lets dealers place product orders through a phone conversation, combining Twilio telephony, the OpenAI Realtime API, and LangGraph for stateful workflow orchestration. The system routes requests through discrete business tools such as search_product(), check_inventory(), and create_draft_order(), backed by a FastAPI service with PostgreSQL and Redis, before pushing confirmed orders into an ERP. The developer emphasizes that the LLM is not treated as the source of truth, with real-time voice chosen to avoid the multi-second latency of a conventional speech-to-text pipeline.

I Built an AI Voice Sales Agent — Here’s the Architecture Behind It What if a dealer could simply call a number, tell an AI what they need, and place an order through a normal conversation? No app. No searching through products. No filling out forms. Just talk. That was the idea behind a project I've been working on: an AI Voice Sales Agent for dealers. For example: "I need 50 boxes of Product X and 20 of Product Y." The AI should understand the request, check whether the products are available, figure out the dealer's pricing, apply any applicable schemes, confirm the order, and eventually push it into the company's ERP. This isn't a finished enterprise product yet. I'm still building and figuring things out, but I wanted to document how I'm approaching the architecture and some of the decisions I've made along the way. The Architecture The basic flow looks something like this: Dealer ↓ Phone Call ↓ Twilio ↓ OpenAI Realtime API ↓ LangGraph ↓ Business Tools ↓ FastAPI Backend ↓ PostgreSQL / Redis ↓ ERP The interesting part isn't just getting an LLM to talk. The AI actually needs to do things. It should be able to: Let's break down how I'm thinking about each part. The first challenge is simple: How does the dealer actually talk to the system? I'm using Twilio as the telephony layer. The basic flow is: Dealer calls ↓ Twilio receives the call ↓ Audio goes to the AI system ↓ AI processes the conversation ↓ Response is generated ↓ Dealer hears the response Initially, I thought of voice as simply: Speech → Text → LLM → Text → Speech But once you start thinking about an actual conversation, latency becomes a huge deal. Imagine saying something to an AI and waiting 4–5 seconds for every response. Technically, it works. As a conversation? Not great. That's why I'm looking at real-time voice capabilities rather than treating the system like a normal chatbot with a microphone attached to it. For the conversational intelligence, I'm using the OpenAI Realtime API. But here's something I realized pretty quickly: The LLM shouldn't be responsible for everything. For example, if a dealer says: "Give me 50 of the blue ones." The AI needs to understand what "blue ones" refers to. That requires conversation context. The system might have something like: Dealer: ABC Distributors Current conversation: Product: Product X Variant: Blue Requested quantity: 50 Dealer preferences: Preferred warehouse: Pune Preferred language: English So the AI understands the conversation instead of treating every sentence as an isolated question. This is probably one of the parts I'm most interested in. I don't want the LLM to directly interact with my database and randomly decide what to do. Instead, I'm giving the agent specific tools. search product check inventory get dealer price get scheme get customer history create draft order So a conversation could look something like: Dealer: "I need 50 units of Product X." AI: "Let me check the availability." ↓ check inventory "Product X" ↓ Inventory: 72 units available ↓ AI: "We have 72 units available. Would you like me to create the order for 50?" ↓ Dealer: "Yes." ↓ create draft order ... This is where LangGraph becomes useful. Instead of having one massive prompt trying to handle the entire business workflow, the agent can move through different states and use specific tools. Conceptually: START ↓ Understand Request ↓ Identify Product ↓ Check Inventory ↓ Check Dealer Pricing ↓ Apply Scheme ↓ Confirm Order ↓ Create Draft Order ↓ END This also makes debugging much easier. If something goes wrong, I can ask: Which step failed? Instead of: Why did the AI randomly do that? One thing I definitely don't want is for the AI to become the source of truth. The LLM can understand things. It can reason. It can communicate. But it shouldn't be the database. The system needs structured data for things like: Dealer ├── Contacts ├── Addresses ├── Credit Limit ├── Language Preference └── Preferred Warehouse Product ├── Product Variant ├── SKU ├── Pricing └── Inventory I'm using PostgreSQL for this persistent data. I'm also using UUIDs for primary keys and keeping things like timestamps, foreign keys, indexes, and soft-delete support in the domain design. I'm still refining the schema, but getting this foundation right is important because adding AI on top of a messy data model isn't going to magically fix it. Not everything needs to be stored permanently in PostgreSQL. A voice conversation can have a lot of temporary state. That's where Redis comes in. The mental model I'm using is basically: PostgreSQL = Persistent business data Redis = Fast temporary state + caching Let's say a dealer says: "I need 100 units." The AI can't just say: "Sure, I've placed the order." It needs to check what's actually available. Something like: Dealer Request ↓ Identify SKU ↓ Inventory Service ↓ Available Quantity ↓ AI Response If the system only has 60 units, the AI should say: "We currently have 60 units available. Would you like me to create the order for 60?" This is a principle I'm trying to stick to throughout the project: The AI should reason about business data, not invent business data. This is another place where a normal chatbot approach isn't enough. In B2B, everyone doesn't necessarily get the same price. Different dealers might have: So the flow could look like: Base Product Price ↓ Dealer-specific Price ↓ Applicable Scheme ↓ Discount ↓ Final Price But here's an important architectural decision: The LLM shouldn't calculate the final business-critical price itself. The backend should do that. The AI can explain the result to the dealer. The actual calculation should come from deterministic business logic. This is probably one of the coolest parts of the idea. A useful sales agent shouldn't feel like it has amnesia after every phone call. Suppose a dealer usually orders a particular product or prefers a particular warehouse. The system could remember useful information such as: Then a future conversation could be much smoother. Instead of asking: "Which warehouse do you want?" every single time, the system could already know the dealer's preferred warehouse and simply confirm it when needed. But there's an important distinction here. Not everything the dealer says should automatically become permanent memory. Memory needs rules. Some information is temporary conversation context. Some information is actual customer data. And some information probably shouldn't be stored at all. That's something I want to handle carefully as the project develops. Eventually, the AI needs to connect with the systems that the business already uses. That's where ERP integration comes in. The architecture I'm aiming for is: AI Agent ↓ Backend ↓ Business Logic ↓ ERP APIs ↓ Orders / Inventory / Customers I don't want the AI agent directly modifying ERP data. Instead: AI ↓ Tool ↓ Backend Validation ↓ ERP That gives us a much safer boundary between the unpredictable nature of AI and the deterministic nature of enterprise systems. This is probably one of the biggest things I've learned while designing this. It can be tempting to just give an LLM database access and say: "Do whatever you need." But for a system dealing with real orders, pricing and customer information, that's a very bad idea. I'd rather have: ❌ LLM → Database ✅ LLM → Tool → Backend → Database LLM ↓ get inventory "SKU123" ↓ Backend validates request ↓ Database query ↓ Structured result ↓ LLM Now I have a proper place for: And most importantly, I know exactly what the AI is allowed to do. The Tech Stack Frontend: Next.js Styling: Tailwind CSS + shadcn/ui Backend: FastAPI Database: PostgreSQL Cache / State: Redis ORM: SQLAlchemy Migrations: Alembic Voice: Twilio Real-time AI: OpenAI Realtime API Agent Orchestration: LangGraph Vector Search: pgvector Version Control: Git + GitHub Monitoring: Sentry + OpenTelemetry I'm trying to avoid choosing technologies just because they're popular. I want every piece of the stack to have a reason for being there. What I'm Still Figuring Out The project is still a work in progress, so there are a lot of things I'm actively figuring out. A voice agent has to feel like a conversation. Even a technically correct answer feels bad if it takes too long. The agent absolutely cannot randomly invent: Those things need to come from actual systems. What happens when the dealer says: "No, not that one. The other blue one." The agent needs enough context to understand what they're referring to. What if the inventory service is down? The AI shouldn't pretend that everything worked. It needs to understand that the tool failed and communicate that properly. Once you're dealing with actual business transactions, security becomes a major part of the architecture. Things like: can't just be an afterthought. The Biggest Thing I've Learned The biggest lesson from this project so far is: Building an AI application isn't the same as putting an LLM inside an application. The LLM is only one part of the system. A useful AI product needs: AI + Business Logic + Data + Tools + State + Security + Observability + Reliable Infrastructure The model provides the intelligence. But the architecture around it provides the reliability. And I think that's an important distinction, especially when moving from AI demos to actual products. What's Next? Right now, I'm focusing on building the foundation properly before trying to make everything "smart." The next things on my list are: The end goal is pretty simple: A dealer should be able to pick up a phone and complete a business transaction through a natural conversation. Call ↓ Talk ↓ Confirm ↓ Order That's the experience I'm trying to build. Final Thoughts This project has also changed the way I think about software engineering. Earlier, I mostly thought about applications as: Frontend + Backend + Database Now I'm asking a lot more questions: What should the AI be allowed to decide? What should the backend decide? Where does the actual source of truth live? What happens when the AI is wrong? What happens when a tool fails? How do we recover from a misunderstood request? How do we make the whole thing observable? And honestly, I'm still figuring out many of these answers. That's probably my favorite part of building this. I'm not trying to pretend I have the perfect architecture figured out. I'm building it, breaking things, learning, and improving it as I go. Build → Break → Learn → Improve. If you're also building AI agents, voice applications, or AI-powered SaaS products, I'd love to hear what you're working on and what problems you've run into. Let's learn from each other. 🚀