TypeSafe’s Jev ‘even more susceptible to attack’ than LLMs, says security vendor

HiddenLayer researchers reported Wednesday that TypeSafe's Jev model is more vulnerable to adversarial text attacks and prompt injection than traditional LLMs, with the TokenBreak attack succeeding against Jev "out of the box" while normally failing on larger LLMs, according to Kasimir Schulz, HiddenLayer's director of security research and co-author of the blog post. TypeSafe, which rolled out Jev last month and categorizes it as the first "System One" model built for machine-to-machine use, did not respond to multiple requests for comment. HiddenLayer warned that System One models should be treated with caution because they are susceptible to both autoregressive LLM attacks and text classifier attacks, citing a simulated checkout page where an attacker-controlled advertisement caused a browser agent to click away from the user's purchase despite instructions to treat page text as untrusted.

Although the Jev model rolled out last month https://www.infoworld.com/article/4223468/typesafe-ais-new-models-work-with-machines-not-humans.html by TypeSafe claims to have significant advantages over traditional LLMs, a security report published Wednesday argued that its streamlined approach makes it more vulnerable to some kinds of attack, compared to traditional LLMs. “In some cases, Jev is even more vulnerable to attacks, with both adversarial text attacks and prompt injection being able to disrupt the model’s outputs, resulting in incorrect and potentially devastating actions,” said the blog post from researchers at HiddenLayer https://www.hiddenlayer.com/research/system-one-model-security . TypeSafe did not respond to multiple requests for comment. There seems to be debate about whether Jev is technically a large language model LLM , with TypeSafe suggesting it is not, but HiddenLayer saying that it appears to be one. TypeSafe has even coined a name for the category: System One https://typesafe.ai/blog/introducing-system-one-models-and-jev , which TypeSafe described as “a new class of frontier models built to make fast, structured decisions that software can use directly.” Jev is the first—and thus far, only—named product categorized as System One. Jev is a streamlined model designed to function machine-to-machine. Its approach allows it to perform tasks faster and to use fewer tokens. But Kasimir Schulz https://www.linkedin.com/in/kasimir-schulz/ , the director of security research at HiddenLayer, the blog’s co-author, said that the accelerated approach robs Jev of some of the checks and balances that exist in traditional LLMs. This means that Jev can fall for some attacker tricks that current LLMs typically detect and refuse, he said in an interview. Despite this, “Jev is great. You should use it. It has a lot faster response time,” he said, with the caveat “it doesn’t have all of the protections” seen in traditional LLMs, so some attacks on it succeed that should have failed. “ TokenBreak https://www.hiddenlayer.com/research/the-tokenbreak-attack , for example, normally does not work with larger LLMs, but Jev is vulnerable to it out of the box,” Schulz said. Although simply inserting characters in front of important words, as in the original TokenBreak attack, did not yield much, HiddenLayer’s blog noted, “we did notice that increasing the number of perturbations and improving their targeting led to much higher attack success rates.” But although traditional LLMs are typically able to see through such attacks, it pointed out, “System One models appear to be vulnerable. As a result, this class of decoder should be treated with caution, given that they are susceptible to both autoregressive LLM attacks and text classifier attacks, providing potential adversaries with greater optionality.” The post cited some examples: “On a simulated checkout page, an attacker-controlled advertisement caused the browser agent to click away from the user’s purchase and respond to the ad despite instructions to treat page text as untrusted,” it said. “In a separate classification test, targeted TokenBreak-style changes lowered toxicity scores for prompts both models had initially flagged.” The post argued that multiple risks are therefore associated with Jev. “A browser agent may immediately click a selected element. A moderation system may allow an input whose toxicity score falls below a threshold. A routing system may send a request to a different tool or workflow. Each of these turns a model judgment into an operational decision, and therefore, risk,” the post said. “When evaluating System One models, test those decisions in the application where they will be used. Include untrusted webpages, misleading task-related text, and modified inputs that preserve their meaning while changing their form. Monitor the selected actions and scores and put independent checks around outcomes that matter. Keep in mind that, under the hood, these are still language models at heart.” Analysts agreed with both the report’s methodology and its technical conclusions, while also pointing out that a security vendor concluding that a new approach is not secure is somewhat self-serving. Philip Harris https://my.idc.com/getdoc.jsp?containerId=PRF005506 , an IDC research director, observed that the methodology described in the blog post “is sound, and consistent with established AI red-teaming practice: building a controlled test environment and probing for adversarial manipulation and injection is exactly the right approach.” But Gartner VP Analyst Nader Henein https://www.gartner.com/en/experts/nader-henein added that Jev, which was introduced on Sept. 15, is so new that he thinks some of the flaw reports were premature. “Jev will go through the same maturity process as do all other models, including susceptibility to seemingly rudimentary manipulation attempts,” Henein said. “This model was only released last month, so being surprised that it has issues is similar to being surprised when your baby drools.” However, Frank Dickson https://www.linkedin.com/in/frankdickson/ , principal analyst at Dickson Research, noted that the same attributes that make Jev so attractive also expose it to riskier behavior. “Jev can only choose from options the developer puts on the menu,” he said. “It limits attack paths. However, if ‘click any element on the page’ is on the menu, the attacker’s ad is on the menu. That is an architecture decision, and the buyer makes it.” He pointed out, “Jev makes AI decisions faster and cheaper, which also makes the wrong decisions faster and cheaper. Speed is a feature. It is not a security control.” Independent technology consultant Steven Eric Fisher https://www.fisher-mns.com/about/ agreed. “Constraining the format of a decision does not necessarily constrain the correctness, trustworthiness, or consequences of that decision,” he said, noting, “the simulated checkout scenario in the post illustrates how an agent can be redirected from an authorized task by information introduced through a less-trusted component of its operating environment. The fact that the agent was explicitly instructed not to trust page content, yet still responded to the malicious advertisement, is particularly relevant.”