Why self-hosted inference is essential

Red Hat AI warns that enterprises relying on third-party hosted APIs for AI agent inference undermine their own data sovereignty, as every prompt and tool call routes through external datacenters. The company argues that open-weight models running on self-hosted infrastructure have been unreliable for agentic workloads, but its vLLM inference engine, generally available on Red Hat OpenShift AI, aims to close that reliability gap. Red Hat's BYOA (bring your own agent) platform provides production infrastructure for any agent framework without code changes, but requires a self-hosted inference layer to maintain full sovereignty.

Your agent can reach its tools. Identity is scoped. Governance is in place. None of that matters if the model behind the agent cannot reliably call those tools. I have watched this play out. A team builds an agent on LangChain or CrewAI. It works in development. It passes staging. Then someone asks the obvious question: where is inference actually running? The answer, almost every time, is a third-party hosted API—OpenAI, Anthropic, or Google. Not because the team prefers it, but because open-weight models running on their own infrastructure weren't reliable enough for agentic workloads. The framework handles orchestration. The hosted API handles inference. Neither is under the organization's control. That creates a sovereignty dilemma. The same enterprise that's committed to data sovereignty—because compliance or regulations require it—is routing every prompt, every tool call, and every reasoning step through someone else's datacenter. In our previous articles /en/blog/why-single-ai-agents-fail-scale-building-governed-multi-agent-networks we discussed 3 failures that hit a single AI agent deployment overnight—43 duplicate tickets, $4,000 charged to the wrong account, and a hallucinated refund policy that led to a $280 return the company had to honor. Those failures happened because production infrastructure was absent—the gap between a working agent in development and a production-ready deployment is an infrastructure problem. If the inference layer also sits outside your control, that gap isn't operational, it's jurisdictional. BYOA—bring your own agent—is Red Hat AI's approach. The platform provides production infrastructure for any agent framework without code changes. But that story starts to crack if the inference layer undermines the very sovereignty the platform protects. This article covers how Red Hat helps close that crack. Chat is forgiving. Agents are not. Most enterprises started with chat. A user asks a question. The model answers. Single turn. Stateless. If the response is slightly off, the user rephrases and tries again. Graceful degradation is built into the interaction pattern. Agentic workloads break that assumption in every dimension. An agent making a multiturn decision—planning steps, calling tools, evaluating results, revising its approach—needs the model to format tool calls in exactly the schema the tool expects. One malformed JSON object in a 6-step tool call sequence does not produce a vaguely wrong answer. It breaks the entire workflow. The tool call fails. The agent retries creating 43 duplicate tickets, if you recall the 6 AM incident . Or worse, the agent silently produces incorrect output—like telling a customer the refund window is 90 days when the actual policy is 30. Think of it this way: chat is a conversation. Agents are contracts. A conversation tolerates ambiguity. A contract does not. When the model generates a tool call, it isn't suggesting an action—it's executing one. The precision threshold is categorically different. This is why most enterprises default to hosted frontier models for agentic workloads. The reliability gap between the best hosted models and open-weight models running on-premise has been wide enough that the sovereignty compromise felt necessary. For a decision maker, that compromise means every agent interaction generates data in a jurisdiction you don't control. For a developer, it means a deployment architecture that can't survive an API deprecation notice. What vLLM changes That reliability gap isn't permanent. It's an engineering problem—and Red Hat is systematically closing it. vLLM Red Hat AI's self-hosted inference engine is generally available on Red Hat OpenShift AI. It's the foundation of the agentic stack. If inference doesn't perform for agentic workloads, everything above it—identity, governance, connectivity—loses its foundation. The unreliability of open-weight models for tool calling has had 3 specific causes, and Red Hat is addressing each one. Smart defaults solve the configuration problem. Running an open-weight model for agentic workloads used to require manual tuning—parser selection, temperature settings, tool-calling format. Get any of those wrong and tool calls fail in ways that are difficult to diagnose. vLLM ships opinionated configurations for agentic workloads so tool calling works correctly on first deployment. No manual tuning required. A developer points their agent at the vLLM endpoint and gets reliable tool-calling behavior without becoming an inference configuration specialist. Multiturn hardening solves the compounding error problem. In a multiturn agentic loop—plan, call, evaluate, revise—small token-handling inconsistencies across turns accumulate. By the 4th or 5th turn, the model is operating on degraded context. vLLM's multiturn hardening reduces those accumulated errors across complex agent workflows. For decision makers, this is the difference between an agent that works in a demo 1-2 tool calls and one that works in production dozens of calls across minutes or hours . Model stratification matching model capability to task complexity solves the cost and latency problem. A capable orchestrator model handles planning and coordination. Smaller specialist models handle execution tasks—tool calling, data retrieval, code generation. I find this is the concept that clicks fastest with decision makers: you aren't choosing between one expensive model and one cheap model. You are running the right model for each task in the workflow. All on-premise. All under your control. vLLM's benchmark results track against hosted frontier models as each phase of hardening ships. The goal isn't theoretical parity—it's the specific reliability threshold that enterprise agentic workloads require. Writing code beats writing JSON Smart defaults and multiturn hardening improve how models generate tool calls. But there's a deeper fix—one that changes the format of tool calls entirely. JSON tool calls require the model to generate a precisely formatted schema for every tool invocation. The format is unforgiving. A missing bracket, a wrong type, or an extra field breaks the call. When tool schemas change—and they do—the model must generate the updated structure perfectly. There's no partial credit. Code-as-Actions https://www.anthropic.com/engineering/code-execution-with-mcp an approach where agents write Python code to invoke tools, rather than generating JSON tool-call schemas sidesteps the problem entirely. Instead of asking the model to produce a rigid JSON structure, the agent writes Python code to call the tool. This plays to what open-weight models are already strong at—code generation—instead of forcing them into a format where small deviations are fatal. I didn't expect this pattern to emerge as fast as it did. Cloudflare, Anthropic, and Pydantic independently converged on Code-as-Actions. When 3 organizations with different code bases, different model choices, and different use cases arrive at the same architectural conclusion, that's a signal worth taking seriously. The convergence also validates the approach for decision makers: this isn't a Red Hat experiment. It's an industry direction. For developers, Code-as-Actions eliminates parser maintenance. No custom code to handle different tool schemas. No brittleness when schemas change. For the sovereignty story, Code-as-Actions is how open-weight models close the remaining reliability gap with hosted frontier APIs—without requiring organizations to route sensitive data off-premise. Owning the API contract, not just the model weights Reliable inference and reliable tool calling solve the performance problem. But there's another lock-in risk that most teams don't see until they are committed: the API contract itself. An agent built against the OpenAI Responses API https://developers.openai.com/api/reference/responses/overview depends on OpenAI's schema, behavior, and availability. The developer wrote the agent to speak OpenAI's protocol. If OpenAI changes pricing, deprecates an endpoint, or alters a behavior, the organization absorbs that change—on OpenAI's timeline, not its own. Migrating to a different back end means rewriting the agent's integration layer. For organizations running dozens of agents, that rewrite isn't a weekend project. Open Responses Open GenAI Stack OGX is Red Hat AI's server-side agentic loop, formerly known as Llama Stack, that implements the OpenAI Responses API specification addresses this directly. Agents built against the OpenAI Responses API can point at an OGX endpoint with minimal code changes. Same API contract. Different back end. The model running behind the endpoint can be Llama, Granite, or any other open-weight model running on your infrastructure. The portability principle extends across API surfaces. vLLM already provides conformant OpenAI Chat Completions endpoints. Open Responses adds state management and tool-calling in the Responses API format. Anthropic Messages API https://docs.anthropic.com/en/api/messages and Google Interactions API https://ai.google.dev/api/interactions-api front-end support are on the roadmap—organizations that use Claude Agents or Google ADK will be able to point their agents at self-hosted infrastructure without rewriting agent code. For teams that want managed infrastructure with built-in rate limiting and policy enforcement, Models-as-a-Service https://www.redhat.com/en/topics/ai/what-is-models-as-a-service MaaS, currently in technology preview provides a third inference path—a managed model-serving platform on OpenShift AI with Gateway API https://developers.redhat.com/articles/2025/12/09/integrate-openshift-gateway-api-openshift-service-mesh testing and verification and Connectivity Link https://www.redhat.com/en/technologies/cloud-computing/connectivity-link integration. Together, vLLM direct on-cluster , OGX agentic loop , and MaaS managed give organizations a spectrum of control from fully self-managed to platform-managed, all behind OpenAI-compatible APIs. For a hands-on comparison of all 3 paths, see Deploying agents with Red Hat AI: The curious case of OpenClaw https://developers.redhat.com/articles/2026/04/14/deploying-agents-red-hat-ai-openclaw on Red Hat Developer. I want to be precise about what this is and what it isn't. OGX implements wire-compatible APIs from OpenAI, Anthropic, and Google—proprietary vendors, not open standards bodies. This isn't sovereignty over the API specification itself. it's sovereignty over the back end: your infrastructure, your models, your data. The API contract becomes portable. The organization controls when and whether to absorb upstream changes. For a decision maker evaluating vendor risk, that distinction—controlling the implementation without controlling the standard—is the practical form of technology sovereignty that matters. What sovereignty actually looks like at the inference boundary Consider what the hallucinated refund policy from the 6 AM incident looks like through the inference lens. The agent fabricated a 90-day refund window when the actual policy was 30 days. A customer returned a $280 product on day 47, citing the agent's response. The team honored the return. Legal exposure followed from an AI agent making unauthorized contractual representations. Now compare 2 scenarios. In the first, the agent runs against a hosted API. That fabricated output and the customer data in that conversation—the customer's name, their purchase history, the product details—all traveled to a third-party datacenter and back. The failure generated data in a jurisdiction the organization doesn't control. Incident response requires coordination with the API provider. Audit trails depend on what the provider retains and shares. In the second scenario, the agent runs against vLLM on the organization's own infrastructure. The failure still happened—self-hosted inference does not prevent hallucinated output that's what guardrails at the inference boundary address, as covered in "Why prompt-level guardrails aren't enough" . But the failure happened inside the organization's perimeter. The prompt, the fabricated output, and the customer data never left. The audit trail is complete. Incident response follows internal procedures. Regulatory reporting stays within the organization's control. That's what data sovereignty means at the inference boundary. Not preventing failures—preventing failures from generating data you can't account for. Confirmed Sovereign Support https://www.redhat.com/en/blog/red-hat-confirmed-sovereign-support generally available in the U.S. and EU extends that principle even to the support chain. When an organization needs help with an inference issue, the support interaction itself doesn't cross jurisdictional boundaries. Sovereignty isn't a product feature. it's a property of the entire stack—where inference runs, who controls the API contract, how credentials flow, what the audit trail captures. When you control the inference layer, you aren't just running a model, you're deciding where your data lives, which laws apply, and who can access it. That's a different conversation than, "which model is cheapest." The inference layer is reliable. The sovereignty story is coherent. But there's a question neither of those answers: How do you know if the agents running on this infrastructure are actually working? We'll look at that in our next article. Get started Ready to run inference on your own terms? Try OpenShift AI free in the Developer Sandbox https://www.redhat.com/en/technologies/cloud-computing/openshift/openshift-ai/dev-sandbox/trial : Deploy and test vLLM inference in a preconfigured environment Explore Red Hat AI interactive demos https://www.redhat.com/en/interactive-experiences : Serve open-weight models with OpenAI-compatible endpoints Read Red Hat AI documentation https://docs.redhat.com/en/documentation/red hat ai/3 : Smart defaults, model stratification and tool-calling configuration Explore Red Hat AI for developers https://developers.redhat.com/products/red-hat-ai : Trials, demos and upstream project links for vLLM, OGX and Granite Start the OpenShift AI learning path https://developers.redhat.com/learn/openshift-ai : Hands-on tutorials from model serving to inference optimization Take the Digital Sovereignty Readiness Assessment https://www.feedback.redhat.com/jfe/form/SV 1LGSam42lrM4Ddk : Establish your organization's sovereignty baseline Resource Get started with AI Inference About the authors With over thirty years in the software industry at companies like Sybase, Siebel Systems, Oracle, IBM, and Red Hat since 2012 , I am currently an AI Technical Architect and AI Futurist. Previously at Red Hat, I led a team that enhanced worldwide sales through strategic sales plays and tactics for the entire portfolio, and prior to that, managed technical competitive marketing for the Application Services middleware business unit. Today, my mission is to demystify AI architecture, helping professionals and organizations understand how AI can deliver business value, drive innovation, and be effectively integrate into software solutions. I leverage my extensive experience to educate and guide on the strategic implementation of AI. My work focuses on explaining the components of AI architecture, their practical application, and how they can translate into tangible business benefits, such as gaining competitive advantage, differentiation, and delighting customers with simple yet innovative solutions. I am passionate about empowering businesses to not only harness AI to anticipate future technological landscapes but also to shape them. I also strive to promote the responsible use of AI, enabling everyone to achieve more than they could without it. More like this Make every GPU-hour count: Progress tracking in Red Hat OpenShift AI /en/blog/make-every-gpu-hour-count-progress-tracking-red-hat-openshift-ai Mastering the AI era: Integrating frontier operations into your technology operating model /en/blog/mastering-ai-era-integrating-frontier-operations-your-technology-operating-model Technically Speaking | Defining sovereign AI with open source /en/technically-speaking/defining-sovereign-AI Technically Speaking | Inside open source AI strategy /en/technically-speaking/open-source-AI-strategy Browse by channel Automation /en/blog/channel/management-and-automation The latest on IT automation for tech, teams, and environments Artificial intelligence /en/blog/channel/artificial-intelligence Updates on the platforms that free customers to run AI workloads anywhere Open hybrid cloud /en/blog/channel/hybrid-cloud-infrastructure Explore how we build a more flexible future with hybrid cloud Security /en/blog/channel/security The latest on how we reduce risks across environments and technologies Edge computing /en/blog/channel/edge-computing Updates on the platforms that simplify operations at the edge Infrastructure /en/blog/channel/infrastructure The latest on the world’s leading enterprise Linux platform Applications /en/blog/channel/applications Inside our solutions to the toughest application challenges Virtualization /en/blog/channel/red-hat-virtualization The future of enterprise virtualization for your workloads on-premise or across clouds