# Xint researcher tells LDS how he breaks AI products > Source: > Published: 2026-08-11 13:53:11+00:00 # Xint researcher tells LDS how he breaks AI products Juno Im, an offensive security researcher at Xint whose team has won at DEF CON, Pwn2Own and DARPA's AIxCC, told Lets Data Science that the first thing he tries against an application with a language model in it is not a jailbreak but a mapping of everything the model is wired to do, then input that makes it do one of those things on his behalf. In an email interview he said that approach has produced refunds nobody approved, access to other customers' accounts and command execution on servers, because applications treat model output as an instruction from an already-authorized user. He named two controls that would have stopped almost every real incident his team has seen. **Juno Im** breaks software for a living, and he is good at it. An offensive security researcher and engineer at **Xint**, he was part of the hacking team behind a record-setting run at DEF CON and Pwn2Own, with wins at DARPA's AIxCC and at Zeroday.Cloud from Google and Wiz. He has helped find high-severity flaws in software trusted by billions of people, including Linux, Android, iOS, PostgreSQL, Redis and MariaDB. Lets Data Science put a set of questions to him about the security of AI systems, a subject that turns attacks on models into attacks on the products data teams ship. ### What AI scanners catch that older tools cannot Im has urged teams to run AI code scanning at enterprise scale. Asked what those tools find that classical static analysis cannot, he drew the line around context. "Traditional scanners only look for patterns that someone wrote a rule for in advance. An AI scanner actually reads the code, so it catches vulnerabilities that only make sense in context: broken business logic, or a permission check that is missing on one endpoint out of several hundred," Im told Lets Data Science. "It also opens old, forgotten code that no human reviewer bothers with anymore." The same context is where they break. "They can only reason about what is syntactically in front of them," he said. "A scanner will often read a single function, see no permission check inside it, and report that as a bug, because whether that function is supposed to be protected, and whether the check already happened upstream in middleware, isn't something the code itself declares. And that's exactly where the noise comes from." ### The first thing he tries against a product with an LLM in it When Im attacks an application with a language model wired into it, he does not reach for the jailbreaks that make headlines. He goes after what the model is connected to. "Prompt injection, though not the jailbreak kind that gets a model to say something it should not," he said. "We map what the model is wired up to do: query a database, call a payment API, send a request to an internal service, run a shell command. Then we write input that makes it do one of those on our behalf." The outcomes are not theoretical. "That has produced refunds nobody approved, access to other customers' accounts, and command execution on the server itself," Im told Lets Data Science. "It works often enough that we assume it will, because the application usually treats the model's output as an instruction from a user who has already been authorized." He offered a separate answer for software written by an LLM rather than software that runs one. "We start with the human's original instructions to the AI, before we read any code. Anywhere those instructions were vague, the agent had to guess. Those guesses are usually where the bugs are." ### What the competitions taught him about autonomous hackers Im's team built autonomous vulnerability discovery systems for competitions including DARPA's AIxCC. The lesson he took away was that the model is only half the machine. "An AI agent is a combination of a model and a harness. During AIxCC we showed that the harness is very important," he said. "It determines the context for the AI agent and the set of tools that the AI agent can access. For example, our harness in AIxCC forces the AI agent to look at every line of code whereas a harness designed for coding will only look at a subset of the code." ### Whether AI auditing favours the attacker or the defender Because AI-assisted auditing arms both sides, Im's answer on open versus closed source depends on the time horizon. In the short term, he said, closing the code can work, and companies are already acting on it. "Cal.com went closed source in April for exactly this reason: if the code is public, an AI scanner can read every line of it, and they decided that was too much exposure for the customer data they hold," he said. "And the fear isn't hypothetical. Metabase is open source, and earlier this month a flaw that let anyone take over an instance without logging in was being exploited before a patch existed." That protection has a short shelf life. "If you ship a compiled binary, scanners are getting good enough at pulling it apart to read it anyway. If you ship a web service and no code at all, they'll find bugs by probing it from the outside," Im said. "And you should be designing on the assumption that your source code leaks at some point, not on the hope that it doesn't." Closing the door also costs something real. "An open codebase is an open invitation to thousands of researchers who will find your bugs and tell you about them, usually for free. Close the door and they stop looking. The attackers don't." ### The two controls that stop most real incidents Asked which controls would have prevented most of the incidents he has seen, Im named two, and put the responsibility on the application rather than the model. "First, check permissions in your own code every time the model uses a tool, against the real user's identity," he said. "Writing do not issue a refund without approval into the prompt is a request, not a rule." The second assumes the worst has already happened. "Assume the machine the LLM-integrated product runs on has already been compromised. Put it in a sandbox, give it the narrowest access it can still work with, and make sure that when something does go wrong, the damage stops somewhere you can live with." His summary left little room: "Almost every real incident we have seen would have been stopped by one of those two." ## Key Points - 1Xint researcher Juno Im told Lets Data Science that attacks on LLM products start by **mapping the tools the model can reach**, not by jailbreaking it, and have produced** unapproved refunds, cross-account access and server command execution**. - 2He argues AI code scanners find context bugs classical static analysis cannot, but generate noise because **they cannot see whether a check already happened upstream**. - 3His two controls: **enforce permissions in your own code** against the real user identity on every tool call, and**sandbox the product as if already compromised**, because a prompt instruction is a request, not a rule. ## Scoring Rationale Exclusive written Q&A provided directly to Lets Data Science by an offensive security researcher from a DEF CON and Pwn2Own winning team, with concrete attack outcomes against LLM-integrated products and two specific controls; directly actionable for teams shipping AI features. ## Sources Original reporting, with the public references used alongside it. LDS Exclusive Reporting based on written answers given directly to Let's Data Science by **Juno Im, security research engineer, Xint**. Practice interview problems based on real data 1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with. [Try 250 free problems](/problems)