We sit down with the two companies to discuss the role of AI in quality assurance, the tech's current limitations, and why human experts are an essential part of the process
Services provider Side has signed a memorandum of understanding with Danish AI firm Modl.ai.
The MOU is binding for six months and will see the two companies working together on an AI-enabled quality assurance system.
Side – formerly PTW – is a long-time player in the services space. Meanwhile, Modl.ai is a newer entrant to the games market. Founded in 2018, the company raised $1.7 million in seed funding two years later.
"The MOU effectively brings the best of what Side has and the best of what Modl.ai has together in a unique solution," Side's chief commercial officer Martin McBride says. "One of the crown jewel, holy grail experiences for the games industry right now has been using AI technology to better augment QA services. No one's really found a great solution for that. We think we have."
Previous AI-enabled QA tech – including that created by Modl.ai – involved developers having to send over entire builds for quality assurance companies for the tech to test. Modl.ai's solution used to be "integration-based" with a plugin and SDK and required developers to provide source code. Unsurprisingly, this created too much friction and "became difficult over time", according to the company's founder Christoffer Holmgard.
This new system, Holmgard says, doesn't require access to any of the internals.
"We take the product as you would give it to a QA tester, as you would eventually give it to the consumer," he says. "You put it on a very similar or even the same device. And then our solution plays the game by looking at the screen, simulating key processes, generating a data set of everything that happens, and then hands it over to a second AI system that has a look at it and identifies bugs, raises it up to the human level and asks whether this is something we should act on."
The first step in the process is what Modl.ai calls building "game knowledge". Developers can upload assets from their project and talk to the AI to build up their 'understanding' of the game.
"All of this is in natural language," Holmgard says. "We just chat with the system about our game, but we're also adding in images, or we might upload our game design documents and so on."
After that, developers define what they want to test. This can vary from "classic test cases" to creating new things that are more "suitable for the AI system".
"This is where you need some human expertise here to adapt to the prompting and the language to the AI agent to make it do the right things," Holmgard explains.
From there, the system generates bug reports based on their session – or run, as Modl.ai calls it – with data produced. "We get all the texts, and we get some video snippets that we call evidence along with all the log data and everything," Holmgard says. "This goes to a human QA specialist, an expert. They review it, and then they decide what happens with it for the rest of our production."
One thing that an AI might find tricky to figure out is whether something in a game is a bug or a feature. Holmgard says that feedback from QA testers will, in part, help the AI system understand the nuances of game design.
"Some of it comes from the prompting. Some of it comes from that game knowledge that you've built up around the game ahead of time," he says. "And then a good part of it also comes from feedback over time from the human testers. And so they will provide feedback to the bug reports and say: 'This was a bug; this was not a bug; this was not a bug for this reason' and so, over time, the system gets attuned."
In fact, Side envisions a future where QA staff move into being AI engineers. In doing so, this will change where quality assurance jobs are done globally.
"One of the cool parts about programs like this is that we get to move our people up the value chain," McBride explains. "So instead of just doing testing and having a career at Side being a tester and moving up to being a senior tester and all the way up, they can become an AI engineer. And so it allows our people to get paid more money. It allows us to have new career paths. And it allows us to potentially bring jobs back to countries and areas where these jobs historically have gone away due to price points and labour concerns. When they're moved back up the value chain, we can move them back into countries where they have historically left."
"This tool really does allow our people to take the best of what AI can offer and reduce the burden and also reduce mistake"
McBride goes on to say that using AI will allow QA testers to have a more "fulfilling career" at Side, as the tech is intended to handle some of the more repetitive and boring work.
"One of the things that this tool does very well right now is that it reduces the burden of repetitive tests," he says. "So for every new build that comes out, you have to do smoke tests. For every new build, there are many verifications. There are so many things that testers have to do that sort of bog down their workflow and don't allow them to think strategically. By kind of knocking these out and automating them, again, they get to have a more fulfilling career here with Side because they get to think more strategically, and they get to actually do more of the analysis and the kind of jobs that help them grow their career here."
Modl.ai's chief product officer Benedikte Mikkelsen adds: "There are some very tedious tests that testers are performing right now. You don't really need a human doing the same flow over and over again. That person could be doing something much more valuable, like assessing the gameplay itself or doing more creative kinds of testing. That, I think, is a value proposition [of using AI]."
McBride also emphasises that AI is not a replacement for QA workers. In fact, he says it will "move people up the value chain".
"There's always going to be an opportunity for people to do a lot of the testing work here because the price point of testing right now is so low in certain countries that it's very hard to augment it completely away," McBride says. "But by giving our people tool sets like this, I'm allowing them to further their career. I'm allowing them to have a longer career outside. And by moving them up the value chain, I get to keep them for longer."
"There are some very tedious tests that testers are performing right now. You don't really need a human doing the same flow over and over again."
McBride adds that Side views AI as a "sliding scale", going on to say that "there will always be a human in the loop".
"That's sort of our ethos," he says. "But this tool really does allow our people to take the best of what AI can offer and reduce the burden and also reduce mistakes. I mean, a lot of this work is tedious and a lot of it. People are prone to making mistakes in human error. You take that out of the equation, and everything kind of just ratchets up a notch and it becomes better."
Asked what type of game this AI solution works with, Side's chief operating officer Oliver Deslandes is candid that the tech right now works with more simplistic titles; that is, not 3D open-world adventures, but rather smaller experiences.
"Really puzzles, card games, stuff that's not too fast," he says. "Stuff that has repeatable mechanics. Anything currently that's very rapid, like fighting games, that's a little further down the line. We have to focus on titles that are more appropriate. What we're aiming for at this stage is really clients that are willing to come on the journey with us because it could very much depends on what they want to do with this. Some of them will be very demanding in terms of wanting results and wanting to save money now."
He concludes: "The reality is that it's a journey, but if you are at the table now with us and maybe doing some pilots with us, then you will see where this tech goes."
Read this next #
Xbox to focus on "efficiency and affordability alongside performance" for next-gen console amid rising hardware costs and innovations in AI - Saber Interactive adds AI disclaimer to Rideshare on Steam; CEO Matt Karch apologises to former lead writer - Testronic appoints Sonia Kerr as CEO, Sharon Baylay-Bell becomes company chair