Generative AI put models at the center of the infrastructure conversation. Agentic AI changes the equation: when agents retrieve data, invoke tools, execute code and coordinate multi-step workflows, the software around the model becomes just as important as inference itself.
To address these evolving infrastructure requirements, Arm has developed Arm AGI CPU, a new class of Arm Neoverse based processors purpose-built for the demands of agentic AI. Arm AGI CPU is designed to deliver the performance, scalability, and power efficiency needed to support highly concurrent agentic workloads, including orchestration, tool execution, data retrieval, and application services.
At OCP Global Summit, Arm introduced a new CPU evaluation methodology with support from Fujitsu, Meta and OpenAI, alongside independently verified results showing Arm AGI CPU delivers up to 2x performance per rack versus leading x86 server processor.
Delivering agentic AI at scale requires more than CPU innovation. It depends on a robust software ecosystem spanning agent frameworks, orchestration, data infrastructure, networking, and application services. Arm enters this new era with a mature, production-proven software ecosystem, built through years of collaboration with open-source communities, software vendors, and cloud providers. With broad native Arm64 support across cloud-native and AI software, and Arm Neoverse based CPUs already deployed at scale across the world’s leading cloud platforms, developers can leverage an established software foundation to build and scale agentic AI applications on Arm AGI CPU.
The software behind an agentic task #
The agentic AI software stack spans agent frameworks and orchestration, tool execution, memory and retrieval, model serving, and cloud-native infrastructure. As illustrated in the chart below, these capabilities are supported by a broad ecosystem of open-source and commercial technologies, including LangGraph, LangChain, Firecracker, Redis, Elasticsearch, and vLLM.
Developers can explore software availability, supported versions, and enablement resources through the Arm Software Ecosystem Dashboard, including its dedicated Agentic AI category. This provides a practical starting point for building and deploying agentic AI applications using familiar technologies on Arm64.
Different agents, different infrastructure demands #
While agentic AI applications share a common software foundation, their infrastructure requirements vary significantly based on the tasks they perform. Coding agents frequently execute code, compile software, and run tests, placing greater demands on CPU performance and sandboxing environments. Enterprise workflow agents rely heavily on API interactions, database transactions, and orchestration, while research agents depend on efficient search, retrieval, and memory management. Personal assistants prioritize interactive responsiveness, and robotics applications introduce additional requirements for simulation and latency-sensitive coordination.
For developers, understanding where an agent spends its execution time is essential to optimizing the software stack. Prioritizing sandbox performance for coding agents, database and API efficiency for enterprise workflows, or retrieval and caching for research agents can improve end-to-end task performance. These differences highlight why agentic AI infrastructure should be evaluated across the complete workflow rather than through model inference performance alone.
Moving beyond enablement to performance optimization #
To address these diverse workload requirements, Arm is focused on optimizing performance across critical components of the agentic AI software stack for both open source and commercial software.
Tool execution and sandboxing: Independent testing of Arm AGI CPU by Signal65 demonstrated up to 45% lower sandbox creation latency compared with AMD EPYC 9965, highlighting the importance of CPU performance for agent tool execution.
Enterprise search and retrieval: Agents rely on efficient access to enterprise information across structured and unstructured data. Elasticsearch supports these workloads through vector search, hybrid retrieval, and real-time search capabilities. Arm’s recent Elasticsearch performance benchmarks demonstrate the benefits of running these workloads on Arm-based infrastructure.
Vector databases and agent memory: Vector databases enable agents to retrieve relevant context and access persistent knowledge across multi-step workflows. Technologies such as Redis support vector search, caching, and agent-state management. Arm is collaborating with ecosystem partners to optimize these workloads on Arm64. Redis 8.10.1 vector search benchmarks demonstrated AWS Graviton5 delivering 16–27% more context fetches per dollar than x86, with a 16% price-performance advantage on a 570K-document dataset under normal load, increasing to 27% on a 3.3M-document dataset under heavy load.
Model serving: Model-serving frameworks are another area of Arm software optimization. Arm’s published vLLM developer resources demonstrate deployment, quantized inference and performance tuning on Arm-based servers, helping developers improve inference throughput and resource efficiency.
Emerging ISVs such as OpenInfer are advancing intelligent inference routing and scheduling across heterogeneous compute resources. OpenInfer is collaborating with Arm to demonstrate its Loom platform running AI models across multiple Arm-based CPUs.
Agentic AI in action: Customer service demo on Arm AGI CPU #
To demonstrate the agentic AI software ecosystem in action, Arm developed an on-CPU, multi-agent customer-service application using LangGraph and LangChain. The application uses a CPU-hosted small language model (SLM) to classify incoming customer requests and route them to specialized agents handling meeting scheduling, customer complaints, product inquiries, and refunds. These agents combine CPU-based inference, enterprise data retrieval, external model APIs, and tool execution to complete multi-step workflows, including scheduling meetings, generating personalized product brochures, and initiating refund processing through enterprise systems. The demonstration illustrates how familiar open-source software components come together on Arm-based infrastructure to enable end-to-end agentic AI applications.
Build and Optimize Agentic AI on Arm #
Arm provides developers with practical resources to build, deploy, and optimize agentic AI applications using the established Arm64 software ecosystem. Through Arm Learning Paths, developers can access hands-on tutorials to build AI agents using llama.cpp, implement RAG applications using LangChain and FAISS, and deploy vLLM model-serving infrastructure on Arm-based platforms. These resources help developers move from software enablement to building and optimizing real-world agentic AI applications.
The growing ecosystem of AI agent skills is also making it easier for developers to build, migrate, and optimize applications on Arm-based infrastructure. AWS has introduced Graviton developer skills to help developers work with AWS Graviton, while Arm is contributing its own skills, including Arm Performix, Arm Performix CI, and Arm Migrate Ease, through OpenAnolis SkillHub. Together, these initiatives bring platform-specific knowledge and optimization guidance directly into AI-assisted development workflows.
Arm continues to collaborate with open-source communities, software partners, and cloud providers to advance the agentic AI software ecosystem. Developers can explore supported frameworks and technologies through the Arm Software Ecosystem Dashboard and use Arm Learning Paths to start building on Arm64 today. Through Arm Learning Paths, developers can access hands-on tutorials to get practical guidance on how to build AI agents using llama.cpp and llama-cpp-agent, execute AI-generated code in isolated Firecracker microVMs, implement RAG applications with LangChain and FAISS, and deploy vLLM model-serving infrastructure on Arm-based platforms.
These resources help developers move from software enablement to building and optimizing real-world agentic AI applications
If you’re developing agent frameworks, tool execution environments, data infrastructure, or model-serving technologies, we’d love to work with you. Connect with the Arm software ecosystem team to explore software enablement, workload validation, and performance optimization opportunities for Arm AGI CPU. And join Arm at AgentCon to experience our agentic AI demos running on Arm AGI CPU. Visit the Arm booth to see how the software ecosystem comes together in a real-world application, connect with Arm’s technical experts, and explore opportunities to build, deploy, and optimize agentic AI solutions on Arm-based infrastructure.
Any re-use permitted for informational and non-commercial or personal use only.