While some organizations are still getting started with their AI strategies, others are in pilot purgatory, with few experiments or proofs of concept (POCs) reaching production. Only 25% of organizations have moved 40% or more of their AI experiments into production, according to The State of AI in the Enterprise. We discussed delivering AI proofs of concept that matter at a recent Coffee With Digital Trailblazers on LinkedIn Live. One key reason POCs stumble is when they don’t align well with the AI business strategy or have defined business outcomes. Two other problems: There isn’t a sufficient AI change management program, or employees aren’t involved in the development process.
But there’s also a significant technology issue: The architecture used for training AI models and developing AI agents can be very different than what’s used for AI inference, running a trained model to generate outputs in production. Training and inference have very different performance, scalability, compliance, and security requirements, and it’s wrong to assume that AI inference is a scaled-up or scaled-down version of the training architecture.
“The industry focus is rapidly shifting from training frontier models to optimizing AI inference in production environments,” says Pascal Jaillon, senior vice president of product at OVHcloud US. “Enterprises are realizing that long-term AI success depends less on raw model size and more on balancing latency, scalability, security, and infrastructure economics across distributed environments. As inference workloads scale, organizations are increasingly evaluating alternatives to traditional hyperscaler-only strategies to improve cost efficiency, data sovereignty, and operational flexibility.”
Optimizing the AI inference environment must also account for operating conditions, compliance requirements, and cost trade-offs. Rick Ross, distinguished technologist at EY, says, “For CIOs, localized inference is a deliberate architectural choice reserved for latency-sensitive applications like robotics, or where regulation mandates.”
Waiting for a successful AI experiment or POC before considering its inference architecture can be a mistake. It may drive unanticipated rework, or add complexities that require restarting the development process. Here are five best practices for developing effective AI inference architecture, infrastructure, and operations.
Training architectures are designed for throughput and flexible data requirements, while inference requires low latency, high reliability, and autonomous operation. Inference environments for AI agents must also consider how workflows will be orchestrated with Model Context Protocol (MCP) servers and agent-to-agent (A2A) integrations.
“IT teams should modernize the integration and orchestration layers first, ensuring they can support event-driven, low-latency, and high-reliability interfaces before AI systems are deployed at scale,” says Riki Efraim-Lederman, division president of Amdocs Studios at Amdocs. “Many legacy environments appear functional because humans are compensating for gaps behind the scenes, but once AI systems begin acting autonomously, that safety net disappears and those weaknesses surface quickly.”
As organizations deploy more AI agents and usage increases, devops teams must consider latency requirements for different use cases and peak-load performance requirements.
“IT teams underestimate how quickly complexity compounds from unpredictable burst traffic, sensitive data pipelines, and AI agents executing across opaque APIs and tool chains,” says Sridhar Iyer, senior director of AI/ML at Versa. “AI inference increasingly requires a distributed architecture, shifting workloads dynamically across cloud, on-prem, and edge locations based on latency, sovereignty, and cost.”
Training environments often require flexibility for accessing multiple large-scale data sources to test and optimize AI models. This contrasts with inference environments, which often connect to fewer runtime data sources and where availability and latency are key design considerations.
“Inference at the edge or across distributed environments only works when the database matches that architecture: local, consistent, and highly available,” says Phillip Merrick, CEO and cofounder at pgEdge. “IT teams tend to treat the infrastructure decision and the data decision as separate workstreams, but they are not, and they are the same decision.”
In training environments, IT can firewall outside access, mask sensitive data, and confine actions to testing environments. A secure-by-design strategy is needed for inference environments where AI agents access real-time data, automate actions across production SaaS platforms, and require dynamic security evaluations around decision-making authorities.
“Inference is the moment a model moves from experimentation into live operations, touching real data, real services, and real business workflows,” says Gal Ordo, cofounder and CPO at Native. “At that point, the critical questions are what the model is allowed to access, what actions it can trigger, and what conditions must always hold while it’s running. Make boundaries explicit from the start, so inference operates inside a controlled, deterministic environment.”
Since AI’s decisions are non-deterministic, observability, auditing, and monitoring are key to avoiding rogue agents, flagging model drift, and alerting early to unexpected usage patterns.
“IT departments need to treat AI inference as another workload with unusual identity, data, and cost characteristics,” says Mike Toole, director of security and IT at Blumira. “It’s critical to choose where it runs based on the sensitivity of what’s going into the prompt and apply the same access controls, logging, and review you’d apply to any SaaS that touches production data.”
Training environments may require GPU chips and other high-performance architectures. For inference, infrastructure needs to focus on compliance, latency, cost, and other non-functional requirements. The differing requirements often result in distinct infrastructures.
“As compute becomes more distributed, CPU is a critical component in inference workloads, and with agents exploding, compute is where they live,” says Michael Reid, CEO at Megaport. “At the same time, inference acts as a north-south traffic multiplier, significantly increasing data transfer demands and putting greater pressure on networking capacity. Fully optimizing for AI inference therefore requires a unified environment where compute, network, and storage work in lockstep.”
Web systems optimized performance by including a caching layer. In AI inference architectures, caching also reduces redundant computation and the associated GPU cost,
”Every request that reprocesses the same inputs from scratch burns GPU cycles at full cost,” says Junchen Jiang, cofounder and CEO at Tensormesh. “Key value caching eliminates that redundancy, cutting latency and GPU spend dramatically. IT teams that build caching into their inference architecture from the start will be able to scale without the runaway infrastructure bills.”
Large enterprises will need to consider hybrid infrastructure based on compliance and performance requirements. For example, AI agents and applications that involve human safety will need to evaluate edge and on-prem infrastructure, while back-office operations may run entirely on public clouds.
“The biggest mistake companies make with AI inference is treating it like a model decision when it’s really an operating model decision,” says Andrea Malagodi, CIO at Sonar. “Where inference runs, whether it be in the cloud, on-prem, or at the edge, directly affects latency, cost, data exposure, and resilience.”
Inference architectures are not built once and then scaled up and down, the way web applications are. Architects should plan for models, infrastructure, security, and data management to all change as technology, compliance, and pricing evolve.
“AI inference is rapidly becoming a core production workload that demands consistent, automated operations across hybrid cloud environments and a clear chain of trust from model to deployment,” says Tushar Katarki, head of product, Gen AI Foundation Model Platforms at Red Hat. “Open source and open standards are essential here; they give enterprises the transparency to secure their AI stack and the flexibility to run inference wherever their business demands.”
One source of change is the AI model capabilities, performance, and costs. Ayaz Ahmed Khan, senior director of engineering at Cloudways, says, “Models are improving at breakneck speeds, and as soon as the model is changed, the prompts and guardrails have to be thoroughly evaluated, reviewed, and modified.”
Another concern is tracking usage and interactions with SaaS platforms, data sources, and other AI agents. Shannon Weyrick, CTO and cofounder at NetBox Labs, says, “IT teams should route AI traffic through a single control point that provides visibility into which models are in use, what data is leaving the organization, and how costs are accumulating, because you can’t secure or manage what you can’t see.”
Matt Waxman, chief product officer at Precisely, says that the most underestimated challenge in enterprise AI inference isn’t the model, but the data behind it. “Prompts and retrieval pipelines pull from dozens of sources with inconsistent semantics, missing lineage, and no governance layer, and the model has no way to know. In an agentic world, where AI systems act autonomously and at scale, that foundation becomes even more critical,” says Waxman.
David Mytton, CEO and founder at Arcjet, shares a practical issue his company encountered with production inference. “Every model wants to become its own API with different request shapes, health checks, metadata, readiness behavior, error formats, and response fields. That does not scale once you have multiple models or back ends,” Mytton says. Arcjet built an abstraction using the Open Inference Protocol on top of their AI security models to provide inference services with a standardized shape for liveness, readiness, metadata, versioned model routes, and tensor-style inputs and outputs.
Organizations moving from dozens to thousands of AI agents will need to advance their finops programs to account for how AI model selection and optimization affect costs.
“As teams move from single-agent prototypes to multi-agent pipelines, inference costs don’t just grow linearly. A multi-agent system can burn 15 times as many tokens as a single chat interaction,” says Andrew Marshall, vice president of product marketing at Yugabyte. “That multiplier is a data problem, not a model one, based on how much context gets passed between agents, how much is retrieved redundantly, and how much state has to be reconstructed from scratch on every call.”
In addition to changing AI models, architects should consider that frontier models used during training may help develop smaller, more efficient models that are then used for inference.
Jason Rolles, CEO and managing director at BlueOptima, says, “LLMs are fast evolving into two broad categories: frontier, cloud-scale models that will likely remain the preserve of hyperscalers, and smaller, highly distilled specialist models deployed at the enterprise edge.” This gives enterprises the option to route low-level, low-ambiguity tasks to smaller models, while tapping frontier models for decisions that require reasoning and accuracy, saving overall costs.
Andrew Filev, CEO and founder at Zencoder, says, “Once agents became useful, usage jumped 10x, contexts ballooned, and enterprises started paying frontier-model prices on every token. Many now burn through annual AI budgets in months, and most of that spend comes from running a flagship model on every step, including simple tasks.”
To increase the number of production AI models and agents, enterprises will need a solid plan for building resilient, scalable inference architectures. But as usage, compliance, technology, and pricing change, plan to reevaluate and evolve the architecture.