What production AI agents actually require Production AI agents require robust state management, idempotency, audit trails, and rollback capabilities, not just fluent LLM calls. A senior engineer argues that most agent demos fail in real workflows because they lack these unglamorous but critical layers, such as persisted state and idempotency keys for external effects. What production AI agents actually require Building agentic AI? I co-run a 6-week cohort where you ship a production-ready agent, not another API wrapper. Most "AI agents" shipping right now are demos wearing production paint. They answer questions fluently and break the moment they touch a workflow with money, state, or consequences. The agent illusion Multi-agent frameworks benchmark beautifully. Five specialist LLMs cooperate, the demo plays cleanly, the README has a diagram with arrows. Then someone wires the thing to a real billing system and it issues three refunds for the same chargeback because a tool call retried on a flaky network. That gap is the actual job most agent tutorials skip. When I review AI code, the same pattern keeps appearing. The LLM call is fine. The agent loop is fine. What is missing is the layer underneath: state, idempotency, audit, and a tool surface the agent cannot use to hurt you. The unsexy layer Systems answer four questions before the agent does anything: What did the agent already do? Persisted state, not "look at the conversation history." What happens if this action runs twice? Idempotency keys on every external effect. Who approved this? An audit log a human can read during a postmortem. Can I roll this back? A clear inverse for every irreversible operation, or a freeze before execution. None of this is glamorous. It is also what separates a system that is a toy demo from one that can run mostly unsupervised in production. The shape of that contract in code: class ExpenseAction BaseModel : idempotency key: str requested by: str requested at: datetime approval required: bool = True dry run: bool = True payload: ExpensePayload def submit action: ExpenseAction, repo: ExpenseRepo - Result: if repo.find by key action.idempotency key : return Result.duplicate if action.dry run: return Result.preview action.plan if action.approval required and not action.is approved : return Result.pending approval repo.persist action return Result.ok action.execute The agent does not call the side effect. It builds a typed plan. A function decides whether to run it. State that survives retries Agents need state management that works across restarts and network failures. The Telegram expense bot we build in our Agentic AI cohort program, uses context.user data to track multi-step flows: python async def handle expense text self, update, context : text = update.message.text result = self. preprocessor.preprocess text if not result.is valid: await update.message.reply text f"Invalid: {result.error}" return ConversationHandler.END response = self. build service .classify result.text .response Store state for the callback handler context.user data "expense description" = result.text context.user data "classification response" = response keyboard = build category confirmation keyboard suggested category=response.category, all categories= c.value for c in ExpenseCategory , await update.message.reply text f"I categorized this as {response.category} {response.total amount} {response.currency} . Confirm or pick another category:", reply markup=keyboard, return ConversationState.WAITING FOR CATEGORY async def handle category selection self, update, context : query = update.callback query await query.answer Retrieve state from previous handler description = context.user data.get "expense description" response = context.user data.get "classification response" if description is None or response is None: await query.edit message text "Session expired. Send expense again." return ConversationHandler.END , category = query.data.split ":", 1 self. build service .persist with category expense description=description, category name=category, response=response, telegram user id=update.effective user.id, await query.edit message text f"Saved as {category} " return ConversationHandler.END The .get with defensive error handling is what saves you when the bot restarts mid-conversation. No silent corruption, no half-written database rows. The user just has to resend their expense description and pick the category again. This is the work of production agents. Tools the agent cannot trust LLMs are undeterministic and hallucinate. Design your tool surface for mistrust: Narrow scopes. read expense and flag expense are two tools, not one tool with a mode flag the LLM can flip. Dry-run by default. Every write tool returns a plan first. The agent opts in to execute. You get human-in-the-loop HITL for free. Schema-validated inputs. Pydantic at the tool boundary /blog/ai-agent-architecture-python/ so a malformed argument cannot reach your database. Explicit confirmation for anything destructive. The agent proposes, a human taps approve. The agent is not the brain of your application. It is a planner that we acknowledge is fallible. The real logic lives in the tools, and the agent's job is to call them with valid inputs and ask for help when it is unsure. Input validation before the LLM sees anything Validate at system boundaries before user input reaches your tools. This prevents XSS, length attacks, and malformed data from consuming tokens: python from dataclasses import dataclass, field import re XSS PATTERNS = "