"Our networks are not generative, so they're not from the same nature as the latest DLSS model"
Today, the British semiconductor and software design company Arm is holding the Arm Everywhere China event in Shanghai to unveil its latest contributions to the AI computing landscape.
The company's aim is to provide a full ecosystem for AI-native computing, including the new AI-native compute platform CSS for Mobile 2, as well as the Arm Mali G2-Ultra NX GPU – a new chip built around neural computing, which promises improved graphics performance at a much lower cost in terms of heat and power consumption. Arm claims its on-device AI will boost graphical performance via a method it positions as more open than Nvidia's divisive DLSS 5 technology, and closer to the developers' original vision.
The firm would not confirm which devices will be using the Mali G2-Ultra NX GPU, but the earlier Mali G1-Ultra chip is found in devices from Xiaomi, Vivo, and OnePlus.
Deyan Lazarov is in Arm's GPU product team as part of the Edge AI business unit, which deals with consumer devices like phones, tablets, laptops, and consoles. He says Arm has been working on the Mali G2-Ultra NX for the past two and a half years, and the thinking behind the chip is to allow mobile developers to take advantage of compute-intensive technology like ray tracing.
"There was actually growing adoption of ray tracing in mobile titles," he says. "The technology has been a bit slow to get going, but now we're seeing more and more titles actually interested in enabling ray tracing."
As GamesIndustry.biz reported earlier this year, the new GPU allows mobile developers to take advantage of Unreal Engine technologies such as MegaLights, and Arm has worked with Sumo Digital to create the game Neural Dawn, which showcases the power of the Mali G2-Ultra NX.
Lazarov says the chip is aimed to support titles that are "mobile first, but then they want to look great: they want to look like desktop titles."
"Then we also have other titles which are just cross-platform by design, so you can play them on pretty much any gaming device that you have. And then those types of titles naturally want to provide a unified experience to their players, so that when you're gaming on your console at home, and then you jump on a train to London or whatever, you don't get a significant drop in the experience."
A few of the games that will be taking advantage of the power of the new chip include Arena Breakout Infinite from Tencent, Where Winds Meet from NetEase, and Infinity Nikki from Infold Games.
Too hot to handle #
"The thing that's different about mobile phones is that they have different thermals and different power budgets from a console," says Lazarov. "They cannot just dissipate heat as much as a console can, as much as a PC can, and they are operating off of a battery all the time."
Arm's solution to this involves integrating neural processing with the more traditional rendering pipe. "It can reduce the GPU workload and it can also reduce the power that the GPU consumes, which then also has knock-on effects on how efficient and capable your device is at the system level."
"So what that basically means is that you get this new efficiency, which helps remove some of the limitations that developers face. And it also helps them not have to worry about making trade-offs as much as they would normally do."
To encourage developers to embrace the possibilities of AI-native graphics, Arm is adopting an open software approach, as well as providing a suite of tools. "We've been working really hard with the ecosystem over the past two years or so," says Lazarov. "The goal that we had was, let's find a way to use AI which is really developer friendly and allows developers to actually have access to these really big efficiency gains – they can then take that extra budget and then reinvest it in visual fidelity if they want to, performance if they want to, battery life if they want to, or a combination of those three things."
The key technologies include Neural Super Sampling (NSS), Neural Super Sampling and Denoising (NSSD), and Neural Frame Rate Upscaling (NFRU). Arm's technology is essentially able to upscale images and cast additional rays, as well as generate further frames using a form of AI reconstruction.
Lazarov gives an example of two frames. "In the first frame, what's happening is that the GPU is actually rendering just a quarter of the image, and then the remaining three quarters are being reconstructed on our neuro accelerators thanks to NSS," he says. "Then in the second frame, there's actually zero work happening in terms of traditional rendering on the GPU. All the pixels are coming thanks to the neuro accelerators, so everything is AI reconstructed."
Not like DLSS 5 #
Nvidia has also been working on AI-enabled technology with DLSS 5, which is able to add extra detail to images – but the technology has been met with controversy owing to the way it can potentially deviate from the developers' original intentions. Lazarov is at pains to point out that Arm's approach is nothing like Nvidia's.
"Our networks are not generative, so they're not from the same nature as the latest DLSS model," he says. "Our networks are convolution type networks, which means that they don't imagine anything, they don't create anything, they just kind of know what things need to look like. So we've trained our networks on real content, but we have also trained them in a way on data sets which allow us to actually make our networks open and public. So all of our networks, our resolution networks, our frame rate generation networks, all of those networks are public and open on GitHub and on Hugging Face.
"What this really means is that a developer can actually get one of our networks and then say, for example, if you have a game which is very quirky because you want to differentiate based on your art style, we give you all the tools that allow you to basically capture data sets from your quirky looking game and then retrain our network, so that our network knows exactly what your game looks like. And when it's upscaling the resolution and inserting new frames, it inserts them in exactly the way in which they should [look] like. And because you are the one who has done that retraining, there's no risk of loss of IP control or anything like that."
"DLSS is not open. It's not like our network. It's proprietary, it's closed, it's a black box, you don't know what's happening"
This approach is very different from the closed-box nature of DLSS 5, says Lazarov. "DLSS is not open. It's not like our network. It's proprietary, it's closed, it's a black box, you don't know what's happening, you don't know what it's trained on. You cannot retrain it on your own [data]. You integrate it, you're happy with the results, or you're not. It's a bit of a take it or leave it, whereas [ours] is take it and make it so that you like it."
Another big difference is that unlike DLSS 5, Arm's technology has been developed with a mobile-first approach, "meaning that our network is designed to operate within every single mobile constraint, be it power, bandwidth – all of those budgets are taken into account."
"If you take something like DLSS, it just wouldn't run on the phone, even if it had software support – you would blow past your budgets instantly."
Hyper-efficient #
Lazarov emphasises how Arm has put a focus on reducing power usage with the Mali G2-Ultra NX. "In addition to these impressive results like 4X performance, 4X efficiency, we also have 70% reduction in the traffic out to DRAM, which is one of the biggest contributors to power consumption on a mobile," he says. "So the less you have to go out to external memory, the less power you burn."
To give an example of the advances of the new chip, he explains that even when pushing last year's GPU to its maximum limit, it could only run Neural Dawn at 40 fps – and only 15 fps under normal conditions. The new GPU, by contrast, can easily run the game at 60 fps.
"In the future we're going to be where the developers are, we're going to make it as easy as we can for them"
In terms of development tools for the new tech, Arm currently has plug-ins available for Unreal, and has a roadmap of expanding to other game engines. "Unity is definitely something that we're looking at, but for now we basically decided to prioritize Unreal." Of note, Arm recently announced a partnership with Unity China.
"In the future we're going to be where the developers are," says Lazarov. "We're going to make it as easy as we can for them. But we also know that many developers use custom game engines, and we cannot provide plug-ins for those custom game engines. But what we do have is an SDK, and the SDK just gives them a very streamlined pathway towards still having a very direct integration."
He adds that developers can take advantage of Arm's AI-native graphics on previous games. "If you take an existing title that is already shipping, and if you want to integrate our solutions into it, that can take you as little as three months to ship a new version that can have these things in it."
Lazarov notes that Neural Dawn was built by a team of fewer than 20 in less than 18 months, and that Arm's tech allows developers to "very quickly and easily make existing games look better, be more efficient, or whatever direction they like to go in – or we just allow them to ship these groundbreaking titles very quickly."