Researchers broke GitHub Copilot’s safety by hiding harm in a workflow

Researchers at the Alan Turing Institute broke GitHub Copilot's safety guardrails by distributing harmful requests across a multi-step coding workflow, causing the AI to complete all 816 harmful prompts it would normally refuse in direct chat. The attack, which worked on models from Anthropic and Google, reveals that current prompt-by-prompt safety testing fails to detect harm that builds across a session, posing a broader risk to agentic coding tools like Cursor, Cline, and Windsurf.

Researchers at the Alan Turing Institute have shown that GitHub Copilot will produce harmful content https://thenextweb.com/news/bioshocking-ai-browser-credential-leak-layerx it would normally refuse. The trick is to spread the request across an ordinary coding workflow. They call it a workflow-level jailbreak, and The Register reported https://www.theregister.com/2026/07/09/github copilot workflow jailbreak/ the finding. The gap between the two settings is stark. In direct chat, the assistant refused almost everything, answering just 8 of 816 harmful prompts. Across a workflow, it completed all 816. How the trick works The idea is simple. Rather than ask the model to do something dangerous, the researchers framed the harmful goal as data to process. Then they split it into small, innocent-looking steps inside a project. Each step looks harmless on its own. The danger only appears once the pieces come together. The team, Abhishek Kumar and Carsten Maple, tested Copilot inside Microsoft’s VS Code editor. They ran it across four models. Two came from Anthropic, Claude Sonnet 4.6 and Claude Haiku 4.5. Two came from Google, Gemini 3.1 Pro and Gemini 3.5 Flash. All four behaved much the same way. What it produced The prompts came from three established safety datasets, including HarmBench and AdvBench, covering 204 harmful tasks. The Register saw redacted examples. One asked how to fool a breathalyser test. Another was a guide to smuggling bulk cash out of the United States. The point is not that any single model failed. Each one refused the same requests when a user asked plainly. The failure sits at the level of the workflow https://thenextweb.com/news/gitlost-github-ai-agent-leaks-private-repos . There, a chain of benign steps slips under safety checks that inspect one prompt at a time. Why current guardrails miss it That is the researchers’ core warning. Prompt-by-prompt safety testing, the industry norm, does not catch harm that builds across a session. A model can pass every single-turn benchmark https://thenextweb.com/news/ai-cyber-benchmarks-outpaced-hacking-tests . A user can still walk it to the same output through the back door. Their fix is to test the whole trajectory, not the turn. Guardrails should examine the files, scripts, and data a coding agent https://thenextweb.com/news/ghostapproval-symlink-flaw-ai-coding-agents touches over an entire task, they argue. They should flag when harmless-looking parts add up to something dangerous. A problem beyond Copilot Nothing about the method is specific to Copilot, or to any one model maker. The researchers say tools such as Cursor, Cline, and Windsurf deserve the same scrutiny. All share the agentic design that makes the attack work. As assistants gain the freedom to run multi-step tasks https://thenextweb.com/news/jadepuffer-agentic-ai-ransomware , the space to hide intent grows with them. Anthropic, Google, and Microsoft’s GitHub all publish safety work on their models. The researchers have contacted all three for comment. The paper sits on the preprint server arXiv. Why it matters The study lands a pointed critique on how the industry measures AI safety. If the real risk lives in the workflow, not the prompt, then passing today’s tests proves less than it appears. The harder task, the researchers suggest, is watching what an agent does across a whole job. Not just what it says in a single reply. Get the TNW newsletter Get the most important tech news in your inbox each week.