AI agent suggested installing a malware package. Engineer almost took its advice

Softjourn, a consulting and software development company, narrowly avoided a supply chain attack when an engineer almost installed a malware package recommended by an AI agent. The company's policy of verifying AI-recommended packages on GitHub revealed the package had few downloads and was created days earlier, exposing a tactic called 'slopsquatting' where attackers register packages under names hallucinated by AI models. Managing director Sergiy Fitsak emphasized the importance of human verification, stating, 'Skipping that step once is how a team ends up explaining a supply chain compromise instead of shipping a feature on time.'

AI agent suggested installing a malware package. Engineer almost took its advice Source: The Register https://www.theregister.com Fortunately, the company had a policy of checking source code on GitHub first PWNED Welcome back to PWNED, the column where we make fun of those who are security self-owned, so hopefully you don’t do the same. This week, we have a story that’s hot off the presses about a company almost sabotaging its security by using AI for programming. Have a story about someone leaving a gaping hole in their network? Share it with us at pwned@sitpub.com. Anonymity is available upon request. Our tale of machine learning /glossary/machine-learning malfeasance comes courtesy of Sergiy Fitsak, managing director of Softjourn, a consulting and software development company. He reminds us that, when it comes to AI, don’t trust: verify. During the course of business, one engineer asked an AI agent /glossary/ai-agent to recommend a package that they needed for a common task. The agent came back with the name of a legitimate-sounding package, which was formatted like a familiar library. At many organizations, this would have been the end of the story. The developer would have taken the AI agent’s advice and downloaded and installed the recommended package. However, at Softjourn, the company has a policy which they actually followed: double-check any software recommendations made by AI to make sure they are legit. The developer skimmed the recommended package’s source code on GitHub and noticed that it had few downloads and had just been created a few days earlier. In other words, it was suspicious. According to Fitsak, attackers have found a way to exploit package names hallucinated by AI models. “The problem is that AI models sometimes invent package names that sound plausible but don't exist, a pattern security researchers have started calling 'slopsquatting,'” he told us. “Attackers have caught on and now register real packages under those exact invented names, betting that a developer under deadline pressure will install first and check later.” If Softjourn hadn’t been so careful, they could have installed a malware package. We don’t know the exact payload, but this malware package could have given crims a backdoor into their systems and the ability to steal data or wreak other havoc. “We caught it because we'd already built a habit of verifying download counts and reviewing source code on GitHub before installing anything an AI recommends, even when it looks routine,” Fitsak said. “It takes a few extra minutes. Skipping that step once is how a team ends up explaining a supply chain compromise instead of shipping a feature on time.” The lesson here is a very simple one: Don’t trust the package names that AI agents recommend. Have a human check the supply chain. And always have a human in the loop so they can take the time to stop and approve any outside code that comes into a project. ®Get AI news in your inbox Daily digest of what matters in AI.