Wiring and powering GPUs differently can swing AI latency by orders of magnitude, says CoreWeave

CoreWeave Inc. senior vice president of AI initiatives Lukas Biewald said at the company's Fully Connected event that GPU networking and power distribution choices can swing AI latency by "orders of magnitude," as CoreWeave launched CoreWeave Forge, a development layer running training, inference, evaluation and agent development in one connected environment. LlamaIndex Inc. co-founder and CEO Jerry Liu said his company's workload is now about 75% inference and 25% training, processing millions of document pages per day for finance, legal and insurance customers while owning no GPU cluster. CoreWeave is expanding beyond GPU compute into networking, storage and software, and Biewald said it follows Nvidia Corp.'s standard networking protocols rather than the proprietary APIs used by hyperscalers such as Amazon Web Services Inc.

Wiring and powering GPUs differently can swing AI latency by orders of magnitude, says CoreWeave The neocloud market is moving past its origins as a stopgap for scarce graphics processing units. AI-native startups now choose their infrastructure on latency, burst capacity and openness, not just chip availability. That shift is playing out at CoreWeave Inc., which is expanding beyond GPU compute into networking, storage and software https://siliconangle.com/2026/09/30/token-economics-reshape-ai-infrastructure-fullyconnected/ as inference demand grows. At the same time, LlamaIndex Inc. has evolved from an open-source framework for retrieval-augmented generation into a model builder that rents its compute rather than owning it, according to Jerry Liu https://www.linkedin.com/in/jerry-liu-64390071/ pictured, right , co-founder and chief executive officer of LlamaIndex. “We’re effectively a specialized AI lab right now that’s purely focused on building models for document parsing and extraction. We post-train open-weight models, we gather our own datasets and we make it really, really good at analyzing and reading documents to basically extract that data,” Liu said. “We care a lot about making sure that we can actually tailor everything we’re doing at the Pareto frontier of performance, cost, and latency for our customers.” Liu and Lukas Biewald https://www.linkedin.com/in/lbiewald/ left , senior vice president of AI initiatives at CoreWeave, spoke with theCUBE’s John Furrier https://www.linkedin.com/in/furrier and Dave Vellante https://www.linkedin.com/in/dvellante/ at Fully Connected https://www.thecube.net/events/coreweave/coreweave-fully-connected-2026 , during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed long-running agents, governance and why AI-native startups are turning to specialized clouds for inference-heavy workloads. Disclosure below. Why bursty AI workloads favor the neocloud model LlamaIndex’s compute footprint barely existed a year ago. Today its workload runs about 75% inference and 25% training, and it processes millions of document pages per day for finance, legal and insurance customers whose paperwork arrives in bursts, according to Liu. The company owns no GPU cluster, so guaranteed capacity matters more than hardware ownership. “We serve a lot of different customers at extremely persistent and also spiky workloads,” ,” Liu said. “We really, really need to make sure that we have the right capacity to serve our customers without getting throttled.” CoreWeave is betting that capacity alone is not the differentiator. Biewald joined the company through its acquisition of Weights & Biases, the AI observability startup he co-founded, and CoreWeave used the event to launch CoreWeave Forge https://www.coreweave.com/news/coreweave-forge-launches-turning-the-ai-loop-production-run-into-a-better-model-and-agent , a development layer that runs training, inference, evaluation and agent development in one connected environment. Coming from software, he initially questioned how much chip configuration could really matter, Biewald noted. “I’ll tell you, the answer is ‘massive difference,'” Biewald said. “I’m talking orders of magnitude difference depending on how you do the networking for the chips and how you do the power distribution.” Openness also separates CoreWeave from the hyperscalers, according to Biewald. Where providers such as Amazon Web Services Inc. lean on proprietary application programming interfaces that make workloads hard to move, CoreWeave follows the standard networking protocols recommended by Nvidia Corp., which brings broader open-source support. Analysts have observed that CoreWeave is broadening its portfolio much as AWS did in its early days https://siliconangle.com/2026/09/26/coreweaves-next-test-from-gpu-scarcity-to-a-durable-ai-cloud/ , even as the company bristles at the neocloud label. “CoreWeave knows that everyone is coming from a different cloud,” Biewald said. “Everyone’s going to host their web service on AWS or GCP, not on CoreWeave. CoreWeave is okay with that, so CoreWeave plays much more nicely with the other clouds.” That ecosystem points to a larger change in who gets to build intelligence, Liu noted. Post-training a small open-weight model remains a skill limited to a narrow group of specialists today. Abundant neocloud capacity, combined with fast-improving coding agents, could open that work to far more people. “Everyone is starting to get really good at defining observability and evals and the right metrics to focus on,” Liu said. “I think there’s going to be a world where we’re basically just going to automate this entire loop and make it accessible to everybody.” Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of Fully Connected https://www.thecube.net/events/coreweave/coreweave-fully-connected-2026 : Disclosure: TheCUBE is a paid media partner for the Fully Connected 2026 event. Neither CoreWeave, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE. Photo: SiliconANGLE A message from John Furrier, co-founder of SiliconANGLE: Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network , where technology leaders connect, share intelligence and create opportunities. - 15M+ viewers of theCUBE videos , powering conversations across AI, cloud, cybersecurity and more - 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/ https://siliconangle.com/aws-marketplace/ About SiliconANGLE Media SiliconANGLE https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fsiliconangle.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=SiliconANGLE&index=9&md5=646b1b564e2259100a2b8638aab0a552 , theCUBE Network https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.thecube.net%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+Network&index=10&md5=7de2a85f95ab4a4a495cede20b8cb1da , theCUBE Research https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fthecuberesearch.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+Research&index=11&md5=7bb33676722925eb57d588ec343e4f6f , CUBE365 https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.cube365.net%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=CUBE365&index=12&md5=d310fb35919714e66ad8d42c9c0c1bc6 , theCUBE AI https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.thecubeai.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+AI&index=13&md5=b8b98472f8071b23ebb10ab9a8dd0683 and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI. Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.