Sponsored #
[Modernization] Across the public sector, technology leaders are working to meet government officials’ pressing demands to integrate artificial intelligence into core operations. Yet despite thousands of AI use cases underway in government, agencies continue to stumble in deploying AI successfully at scale. That’s due to several factors, but one in particular is the misconception about the hardware required to leverage AI — and the frequent failure to distinguish between two phases of AI: training and inference.
There is a pervasive belief that enterprise-level AI workloads require massive, proprietary GPU farms. This misconception has its roots in what industry insiders call “CUDA lock-in” — a reference to a parallel computing model developed two decades ago for Nvidia graphic processing units (GPUs). That coding ecosystem would later prove useful for training massive, large language models, which have since fueled a costly arms race to secure GPUs.
However, the reality is that as much as 90% of enterprise and most government AI workloads use AI for inference — running existing, pre-trained models to extract insights, summarize reports, answer queries, and streamline workflows. Agencies generally do not require Ferrari-class GPU farms. AI inference can accomplish these tasks far less expensively using existing CPUs running in agency data centers — and Neural Processing Units (NPUs) in enterprise laptops — both of which are highly efficient at executing these workloads.
Moreover, the energy economics of these architectures are also vastly different:
- NPUs are specifically engineered to process AI workloads at a remarkably low power envelope — often under 30 watts.
- GPUs, by contrast, consume anywhere from 350 to 1,500 watts per unit, generating massive heat and soaring electricity bills.
For federal leaders, recognizing and avoiding this architectural trap is not merely a matter of technical preference or economic choice; the wrong architecture can make AI difficult to scale, govern, and afford in the long run. The hidden “token trap” of agentic AI
There’s another consideration federal leaders need to come to terms with. As federal agencies move beyond basic chatbots and transition toward autonomous, multi-step agentic AI, they face a murkier economic challenge: the “token trap.”
In a basic search query, the exchange of data (or “tokens”) is predictable and linear. But advanced agentic AI utilizes continuous reasoning loops. When a user enters a single prompt, the system does not just generate one answer. Instead, dozens of back-and-forth automated queries run in the background as different specialized models plan, retrieve data, compare answers, and refine their reasoning.
In a billed cloud-only environment, this background rumination causes token consumption to compound exponentially, resulting in unpredictable and volatile monthly operating expenses. If a single complex workflow starts generating thousands of intermediate background tokens, a seemingly simple task can quickly become financially unsustainable.
That’s why federal leaders need to act now to reclaim the economics of AI.By fully assessing how and where AI models are best suited for enterprise workloads, agencies can avoid over-provisioning hardware, keep infrastructure and token costs under control, and, importantly, preserve greater flexibility in the future.
Right-sizing in action
The U.S. Census Bureau offers a good example of an agency that rightsized its infrastructure for AI, accelerated data collection, and saved millions of dollars in part by bypassing expensive GPUs and using standard CPUs for day-to-day inference. Instead of deploying a trillion-parameter cloud-based frontier model, they used a right-sized, 8-billion-parameter local model to parse complex SEC filings for the economic census, successfully processing massive datasets on cost-effective, standard infrastructure.
Crucially, beyond hardware savings, the Census Bureau designed this system from the start with its people and processes in mind. By pairing the local AI model with human subject matter experts, analyst accuracy rose to 99.9% (up from a typical human baseline of 97%) and significantly surpassed AI’s standalone accuracy of 75%.
In another example, the California DMV used locally deployed AI models to optimize customer wait times. Rather than streaming bandwidth-intensive, highly sensitive raw video feeds of waiting lines back to a central cloud hub, the DMV ran object detection locally at the edge on CPU hardware. This reduced network bandwidth, avoided high cloud egress costs, protected citizen privacy, and delivered real-time queue data.
Aligning technology with organizational readiness
Recent industry studies suggest that 87% of AI projects fail to reach production or deliver their targeted outcomes — underscoring the importance of addressing organizational processes, data readiness and workforce training alongside hardware decisions.
To build a scalable, secure, and economically viable AI posture, federal leaders should take four immediate actions:
- Consolidate and modernize existing datacenters : Maximize the hardware you already own. Modern enterprise CPUs, such as the Intel Xeon 6 with built-in Advanced Matrix Extensions (AMX), can deliver up to a 2x increase in throughput for AI inference workloads without requiring a dedicated GPU. This enables agencies to achieve up to a 10:1 server consolidation, lowering total cost of ownership by up to 52% and restoring sanity to agency energy and footprint budgets.
- Re-engineer processes before deploying technology : AI is a powerful magnifier, not a leveler. If your agency has a broken, inefficient manual process, throwing AI at it will only magnify and accelerate that failure. Focus on training your workforce and re-architecting workflows to support AI-augmented humans first.
- Adopt small language and ‘local-first’ models: Using CPU-based SLMs can deliver computationalcost savings of 55%, while implementing hybrid agentic AI solutions designed for PCs and edge devices (such as IntelSuperClaw ) for sensitive, high-frequency, and repetitive operations canreduce average cloud compute token consumption by up to 70% .
- Prioritize back-office ‘blocking and tackling’: AI represents a tiny fraction of an agency’s overall IT ecosystem, but an oversized amount of attention. That has led many to neglect fundamental back-office and data management infrastructure. Take the time to rebalance your IT portfolio. An advanced AI engine is useless if your legacy backend cannot feed it clean, secure data.
As AI systems gain autonomy and the number of models grows, the government’s success will depend not only on deploying the right AI workloads at mission scale but also on optimizing infrastructure that delivers outcomes across local, edge, data center, sovereign cloud, and public cloud environments.
Read our report, “The new economics of AI for government.**”