From Demo to Production: The Guardrails That Make an AI Agent Safe to Ship

Hook:Most "AI agents" you see on the internet are demos. Here's the single most common

reason they never reach production — and a small, open-source harness that gets past it.

We are past the phase where the hard part of building an AI agent was calling the model.

The hard part now is the 10% nobody talks about: what stops the agent from doing something harmful? I've seen this from both sides — I built and ran a ~25-agent platform in production

The uncomfortable truth: a chatbox that can call 5 tools is not a product. The difference

between a weekend project and a system you can put in front of customers is three things —

and they're all boring, non-glamorous engineering:

So I wrote a tiny harness that keeps these front and center. It's intentionally small — small

enough to read in an hour — because the value isn't in a framework, it's in the pattern.

An agent's output is a prediction, not a promise. Before it ships, you need a check that

it passes your bar. In the harness this is a pluggable QualityGate

— a rule of thumb you swap

with an LLM judge or a test suite:

@dataclass
class EvalReport:
    passed: bool
    score: float
    checks: List[str]

class QualityGate:
    def grade(self, proposal: str, context: str = "") -> EvalReport:
        return self.grader(proposal, context)

The loop refuses to execute if the gate fails:

result.report = self.quality.grade(proposal, f"state={state}")
if not result.report.passed:
    self.approval.log("quality-gate", "blocked", result.report.__str__())
    return result

Notice it logs the block. In production you'd want every blocked attempt in your

observability stack. "We rejected 12% of agent proposals this week" is a real KPI — it means

the gate is doing its job.

This is the one that actually gets enterprises to say yes. When an agent wants to expedite

an order, cancel a subscription, or move money, it should stop and ask a human. Silence is

not consent.

class ApprovalGate:
    def request(self, action: str, detail: str) -> bool:
        decision = input(f"Approve {action}? [y/N] ").strip().lower()
        self.audit.append(AuditEntry(time.time(), action, "human-reviewer", decision, detail))
        return decision.startswith("y")

In the scaffold, marking a tool needs_approval=True

is enough to route it through the gate:

@tool("expedite_order", "Mark an order as expedited.", needs_approval=True)
def expedite_order(order_id: str) -> str:
    return f"PO {order_id}: marked expedited"

And because there's an audit trail, you can always answer "who changed this and why?" —

which is usually the first question a compliance team asks.

Models change every few weeks, and so do prices. Your agent loop should never know which

vendor it's talking to:

class ModelProvider(Protocol):
    def complete(self, messages: List[Message], tools: Dict[str, Any]) -> str: ...

agent = Agent(provider=default_provider("openai"))   # or "deepseek", "qwen", "mock"

A few lines of abstraction mean you can run the same agent on OpenAI, Azure OpenAI, DeepSeek,

or Qwen with a config change — and test the whole thing offline with a MockProvider

that

needs no API key. That's not just engineering hygiene, it's a cost lever and a hedge.

The other huge pattern: don't let an "agent" freewheel over your critical process. Model the

workflow explicitly. Here's a purchase-order exception state machine — the kind of thing that

shows up in every supply chain / ERP copilot:

class State(str, Enum):
    OPEN = "open"; EXPEDITED = "expedited"; CANCELLED = "cancelled"
    EXECUTED = "executed"; CLOSED = "closed"

class POStateMachine:
    def transition(self, target: State, reason: str = "") -> State:
        if target not in TRANSITIONS[self.state]:
            raise InvalidTransition(f"{self.state} -> {target}")
        self.state = target
        return self.state
open ──► expedited ──► executed ──► closed
  │          ▲              ▲
  └──exception──┘  (human approval gate in between)

The agent proposes a transition; the state machine enforces what's legal; a human approves

anything consequential. That's how you get autonomy with control — the exact phrase

enterprises want to hear.

agent = Agent(provider=provider, registry=registry,
              approval=ApprovalGate(), quality=QualityGate())
sm = POStateMachine(po_id="PO-1234")
result = agent.run("PO-1234 is stuck; expedite it.", state=sm)
print(result.audit)

Pull the repo and run it with no API key:

python examples/po_workflow.py --yes

You'll see the quality gate pass, the human approve, the order get expedited, the state move to

expedited

, and the whole thing recorded in the audit trail. That's a real production-shaped

flow in ~300 lines.

If you're building an agent, build the guardrails first. It's the difference between a

demo you screenshot and a system your business actually trusts. And if you don't want to build

it yourself — or want someone who's shipped this at Microsoft scale — get in touch.

Open-sourced under MIT. Fork it, build on it, and say hi.

About the author: I'm Zhaowei Sun, an AI agent / Copilot engineer who's built and run production agent platforms at Microsoft and high-scale systems at Hulu. I consult on agent architecture, and I keep this repo as the scaffolding I wish I'd had.

Tags: AI Agents, LLM, RAG, Microsoft Copilot, Software Architecture, System Design

Suggested platforms: Dev.to | Medium | LinkedIn | Hacker News (Show HN) | 掘金 (translated)

source & further reading

dev.to — original article I spent one day smashing three real open source bugs App-like UX in Next.js 16.3 Domux: a compact open model for smart-home command understanding at the edge