When Qualcomm acquired AI software infrastructure startup Modular for close to $4 billion, it appeared to be a strange fit. Modular has built a hardware-agnostic software stack designed to abstract away the underlying silicon from developers, enabling portability. Qualcomm sells proprietary silicon, so portability wouldn’t necessarily be in its interests. But according to Modular cofounder and president Tim Davis, the acquisition represents a shared belief that the future of AI compute is heterogeneous, and that software is the key to deployment of non-GPU hardware for AI inference.
One of Modular’s original theses, Davis said, is that the future of AI compute would certainly be heterogeneous. Their other theses—that inference was going to be huge, that superintelligence would need both data center and edge hardware, and that software was holding everything back—have all come true.
“If you tie those four theses together across the ecosystem, and then if we were to consider a partner to help us achieve our mission, who ticks all the boxes and who will enable us to have global distribution while still maintaining an open, neutral platform for the world?” Davis told EE Times in an exclusive interview.
Qualcomm has an enormous hardware footprint, from edge to cloud, so it’s certainly aware of the challenges of running AI on heterogeneous silicon, Davis said.
View All “We felt confident in the Qualcomm team and Cristiano [Amon, Qualcomm CEO] and the other leaders at Qualcomm,” Davis said. “The ability to get extreme distribution much faster by having a backer of that scale was an exciting proposition.”
Velocity is critical if Modular wants to become a new open standard for the world, both in the data center and at the edge.
“The folks at Qualcomm are very excited about maintaining an open, neutral standard, being able to continue to work across all silicon,” Davis said. “The heterogeneous nature of compute is such that for everything to be successful, we must achieve that mission.”
Importantly, Modular will remain an independent entity—the team will be maintained and the mission preserved, Davis said.
“We have a shared understanding that we want an open developer ecosystem,” he said. “To have an open developer ecosystem that drives AI globally, you need an open set of infrastructure that developers are excited about, otherwise there won’t be an ecosystem.”
Mojo and Max
Modular’s key products are Mojo, a kernel programming language that is silicon-agnostic, and Max, an open modelling and serving framework that enables heterogeneous scaling. Both Mojo and Max are open source.
Mojo is designed to make it easy to write core infrastructure that’s natively multi-silicon, covering CPU-ASIC, heterogeneous GPU, and cloud-edge silicon use cases. Mojo 1.0, a stable, production-ready version of the language that has been under development for 4.5 years, will launch shortly.
“Everyone wants a multi-silicon future, for a range of reasons,” Davis said. “A lot of the agentic workflow is now CPU-bound, and we’re getting disaggregated inference becoming a big thing.”
Cloud management and orchestration is still an open problem, Davis said, which requires a software solution because of the shift from selling GPU-hours to selling tokens.
“We want a multi-silicon world where the user doesn’t have to care where the tokens are being generated,” he said. “We made a radical statement four years ago: Developers do not care about the hardware. That is more true today than ever. The industry is focused on tokens per second.”
Max builds on Mojo and is similarly multi-silicon by design. Max is already suitable for multi-modal inputs and outputs, where today’s serving infrastructure is predominantly text-optimized. Projects like vLLM and SGLang have forks that are image and video-optimized, but this is already too much fragmentation, in Davis’ opinion.
Modular has done a lot of work on management, observability, MLOps, and cloud optimizations for Max, Davis said. Max is due to go into general availability in a few weeks.
“We will, over time, add training,” he said. “Then you’ll have a stack that is multi-modal, multi-silicon, and enables you to go from training to inference, from edge to cloud. That’s the dream.”
Alternative hardware
Software infrastructure has been holding back deployments of non-GPU hardware at scale so far, he said. Neoclouds finance most of their GPU cluster buildouts using debt instruments, which is fine for Nvidia or AMD hardware, (they have a known depreciation curve and the assets are liquid, which means they can be sold to someone else), but for alternative silicon there are few financing options today, Davis said. Neoclouds are having to fund clusters with equity, which is extremely expensive and limits fungibility across compute.
“If you have a multi-silicon software layer that makes the fungibility of an asset easier, everyone around the world starts to say that the existing compute economy has overpriced software on a certain provider, and it doesn’t mean the underlying compute asset has changed, it just means that fungibility across compute helps rewrite the financial equation for why more silicon can get into clouds, which really matters,” Davis said.
Qualcomm has realized how big an issue software is, knows that a consistent platform is needed to reduce the burden on developers, and understands that an open ecosystem is key to making that happen, Davis said.
Modular’s customers so far are the most sophisticated enterprises and genAI startups, plus Neoclouds and token factories. “We have customers at every level of the stack,” Davis said. “We have hardware customers—people who want silicon brought up—we have neoclouds who are buying that hardware and want to install it in their token factory so they can make bigger margins, frontier model labs who want to sell to enterprises via API, and we have enterprises who don’t care where the tokens come from.”
Frontier labs increasingly like multi-silicon architectures as they become total cost of ownership (TCO)-driven, Davis said.
“Frontier labs want us to enable them to develop on our open infrastructure and give them multi-silicon diversity, which is great for their business, because now they can work with everyone to scale it across multiple layers of the stack, which is really important,” he said. “They then become part of the open ecosystem we’re creating, helping improve our infrastructure but actually improving it for everyone.”
AI kernel generation
EE Times has spoken to a number of chip companies recently that have augmented or even replaced parts of their compilers with AI assistants or agents. Writing a performant compiler for novel hardware has been a huge barrier to entry for these companies and they are embracing AI code generation in their stacks, perhaps making a Claude-based agent, teaching it about the supplier’s hardware, and then letting it write or convert code at the lowest levels.
Davis is more cautious. Compilers need to be deterministic, he said, noting that AI almost by definition doesn’t produce deterministic outputs, and that mechanistic interpretability is still a huge area of research.
“AI code generation is very exciting, but I view it as an inverse pyramid—it’s amazing at building prototypes, it’s amazing at certain application layer stuff, but the lower you go, the more determinism you want,” he said.
Every enterprise will have to decide how much non-determinism they are willing to accept in their code base. For some organizations Modular works with, including financial institutions, any is too much.
“The question is: are you willing to take a risk on production workloads with production customers on vibe-coded stuff at the end of the day?” Davis said.
Simply writing kernels that work is also not enough, Davis said—they have to be comparable or better than CUDA. “Compute portability is still about TCO,” he said. “The fact that AI can write kernels is great, and it’s exciting, but you need to have near state-of-the-art performance for the TCO to work on most alternative silicon today.”
For many reasons, AI code generation still needs a human in the loop, Davis said. “I still think there are people in the world that know that part of the stack incredibly well,” he said. “Above that, we need people who are experts in techniques like speculative decoding… there’s a lot of sophistication in that part of the stack too. And then above that, you need cloud infrastructure to actually orchestrate it all. So it is a whole-stack problem.”
AI-generated kernels aren’t ready for production deployment today, evidenced by existing limitations on commercial stacks, Davis said. The future will need a core substrate to build on, namely, Mojo and Max, he added.
“Once you have a unified programming model at the base that can expose the core intrinsics of your silicon, you get all the AI framework and the serving and everything else just out of the box that you can go and use. That is not true [elsewhere] today, you can’t just go and vibe-code that. It’s a hard systems programming problem to do that and then to tie it all together.”
Modular’s team is excited about convincing the world that the future is heterogeneous.
“It’s inevitable, if you look at history, that we will get there,” Davis said. “But how we get there is an open question. Our strong belief is if we can help all of these companies get there faster by not having to worry about the software as much, that’s going to be wonderful for the world.”
Read also:
[Microchip Acquires Edge AI Chip Startup Hailo](https://www.eetimes.com/microchip-acquires-edge-ai-chip-startup-hailo/)
[Warning Shots Fired as AMD Announces New Data Center GPUs](https://www.eetimes.com/warning-shots-fired-as-amd-announces-new-data-center-gpus-at-advancing-ai-event/)
[Etched Raises $300M with $1B in Pre-Orders](https://www.eetimes.com/etched-raises-300m-with-1b-in-pre-orders/)