Agents are breaking AI economics, and the fix starts in the data layer

Couchbase Inc. expanded its AI Data Plane with native Amazon Bedrock integration featuring semantic caching, a fully managed Agent Memory and Model Context Protocol server on Amazon Web Services, and vector search in its analytics engine now called Operational Insights, senior vice president of AI Tim Finley said in a pre-briefing. Finley cited a Cisco Systems Inc. study finding agents generate 450% more traffic per task than humans doing the same work and Cloudflare data showing machine traffic surpassed human traffic on the web in June, about 18 months earlier than forecast, and said "customers are actively sacrificing intelligence by downgrading models because of the bill once they hit scale." Couchbase claims semantic caching cuts large language model calls by 40% to 60%, Agent Memory reduces token usage by 33% to 45% versus alternatives on the LoCoMo benchmark, and vector-based context retrieval produces prompts about 70% smaller than full-context approaches.

Agents are breaking AI economics, and the fix starts in the data layer When companies talk about the cost of artificial intelligence, the conversation almost always starts with models: which one to use, what it costs per million tokens, and whether a smaller model is “good enough.” That made sense when AI meant a chatbot. It makes much less sense now that agents are doing real work in production. A single agent task can fire off dozens of queries, pull context from several systems, call a model several times and write results back. What a task costs and how fast it runs are increasingly determined by everything that happens around the model. During a pre-briefing ahead of this week’s expansion of the company’s AI Data Plane, Tim Finley, Couchbase Inc. https://www.couchbase.com/ ‘s senior vice president of AI, highlighted a key trend. The company announced native Amazon Bedrock integration with semantic caching, a fully managed Agent Memory and Model Context Protocol server on Amazon Web Services, and vector search in its analytics engine, now called Operational Insights. The more interesting part is the reasoning behind the release, which points to a problem most information technology leaders haven’t fully accounted for. Agents are a different kind of user Finley opened with queuing theory to explain the problem. As load rises, latency grows faster than traffic, and agents quickly drive upload. Finley cited a Cisco Systems Inc. study finding that agents generate 450% more traffic per task than humans performing the same work, along with Cloudflare data showing that machine traffic surpassed human traffic on the web in June, about 18 months earlier than forecast. “Capacity plans built for human growth curves are likely to break,” Finley said. He described agents as “probably the nastiest, most demanding user that you can ever imagine when it comes to running a system.” I think this is the most underappreciated risk in agentic AI. Most production databases were sized for people who click, wait and click again. Agents fan out into parallel tasks, need current and historical data in the same loop, and never sleep. Treat them as “just more users,” and the data tier becomes the bottleneck, usually right after a successful pilot convinces the business to scale. Companies are downgrading intelligence to pay the bill The pre-briefing comment that should concern IT and business leaders was the trade-off between model choice and capabilities. “Customers are actively sacrificing intelligence by downgrading models because of the bill once they hit scale,” Finley said. That matches what I hear from IT leaders. The pilot runs on a frontier model, the results look great, then finance sees the inference bill at production volume, and the team falls back to a cheaper model, degrading the results that justified the project. Often, that trade-off isn’t necessary because much of the spend is waste: the same questions sent repeatedly, bloated prompts,and agents that start every session from scratch. This is where Couchbase’s announcements can help mitigate the tradeoff. Semantic caching serves similar queries without calling the model again, and the company says it cuts large language model calls by 40% to 60%. It also claims Agent Memory reduces token usage by 33% to 45% compared with alternatives on the LoCoMo, or long-term conversational memory, benchmark, and that vector-based context retrieval produces prompts about 70% smaller than those from full-context approaches. These are Couchbase’s own numbers, and results will depend heavily on how repetitive a workload is. But the direction is right: most organizations have more room to cut token waste than to improve model quality. Convergence versus consolidation During the briefing, Finley explained the difference between convergence and consolidation. The latter reduces the number of tools you pay for. With the former, “a converged system is greater than the sum of its parts” because every capability shares a single copy of the data. The vector search in Operational Insights is a good example. Finley walked through a phone manufacturer’s use case to determine which factories are linked to the most positive battery-life reviews. Typically, that involves a relational database, a key-value store, a vector database, an ETL pipeline and a warehouse. Couchbase says it answers it in one SQL query, using “cluster by” much like “group by,” then writes the result back so agents can read it. “There has been a fallacy for a long time with analytics that a really great insight was the finish line,” Finley said. “We think it’s the action.” Every hop between systems adds latency, introduces a sync job that can fail, adds another security model and creates another place where data can go stale. An agent working from stale data can be confidently wrong. Fewer moving parts matter as much for AI speed and accuracy as they do for cost. The competitive picture Snowflake Inc. and Databricks Inc. have both bet on Postgres, and Postgres with pgvector is the default for many AI teams because it’s free and familiar. Finley, who came from AWS’ database business, didn’t discount this value, but he argued that it strains at Fortune 2000 scale once vector counts reach the hundreds of millions. Couchbase also points to deployments at LinkedIn, Amadeus and telecom software provider Mavenir. Brandon Larson, senior vice president and general manager of Cloud, AI & IMS Business Strategy, explained, “Couchbase has been a key component in achieving high availability in Mavenir’s voice solutions for years,” explained Brandon Larson, senior vice president and general manager of Cloud, AI & IMS Business Strategy. “Now we’re creating new AI voice services that require getting reliable context fast, and Mavenir is building that context layer with Couchbase because of their proven performance in production.” Recommendations for IT leaders Model agent load, not user load. Rebuild capacity plans around parallel fan-out, many queries per task and sustained bursts. Load-test the data tier with simulated agent traffic before scaling a pilot. If latency climbs faster than request volume, the queuing effect is at work. Measure cost per task before downgrading models. Find out how many prompts are near-duplicates, how much context is resent each turn, and how often agents rebuild state they already have. Caching, memory, and better retrieval should be the first levers. Switching to a cheaper model should be a quality decision, not a budget one. Count the hops in your agent loop. Map every system and sync pipeline an agent touches to complete a typical task. Each hop is a latency cost, a failure point and a governance gap. Whatever vendor you choose, that map shows where convergence pays off. Treat caches and memory like production data. Semantic caches and agent memory store customer context and model outputs. Apply retention policies, user-level isolation, access controls and audit trails from day one. A cache that returns a “similar” answer to a genuinely different question poses a correctness risk, so tune and monitor similarity thresholds. Insist on proof at your scale. Vendor benchmarks are a starting point. Test at your data volumes, concurrency and vector counts, and check both recall and throughput. What works at 10 million vectors may not at a billion. Keep the memory layer portable. Agent frameworks are evolving quickly. Choose memory services that work across LangGraph, CrewAI, LlamaIndex and other frameworks, so switching frameworks doesn’t require rebuilding the most valuable part of the stack. Final thoughts The industry has spent two years arguing over models. The next two will focus on the cost and reliability of running agents at scale, and much of that will be decided in the data layer. Couchbase is right that agents are a fundamentally different kind of user, and its focus on token economics speaks to the conversation chief information officers are having with chief financial officers right now. Its challenge is winning the architectural argument before customers default to the Postgres-plus-point-tools stack they already know. Zeus Kerravala is a principal analyst at ZK Research, a division of Kerravala Consulting. He wrote this article for SiliconANGLE. Image: Couchbase A message from John Furrier, co-founder of SiliconANGLE: Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network , where technology leaders connect, share intelligence and create opportunities. - 15M+ viewers of theCUBE videos , powering conversations across AI, cloud, cybersecurity and more - 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/ https://siliconangle.com/aws-marketplace/ About SiliconANGLE Media SiliconANGLE https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fsiliconangle.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=SiliconANGLE&index=9&md5=646b1b564e2259100a2b8638aab0a552 , theCUBE Network https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.thecube.net%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+Network&index=10&md5=7de2a85f95ab4a4a495cede20b8cb1da , theCUBE Research https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fthecuberesearch.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+Research&index=11&md5=7bb33676722925eb57d588ec343e4f6f , CUBE365 https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.cube365.net%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=CUBE365&index=12&md5=d310fb35919714e66ad8d42c9c0c1bc6 , theCUBE AI https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.thecubeai.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+AI&index=13&md5=b8b98472f8071b23ebb10ab9a8dd0683 and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI. Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.