Jot I just stumbled upon an excellent 3-parts series of (responsibly disclosed) attacks impacting Microsoft Copilot.
The first one is poisoning Copilot memory via an attacker-controlled webpage, which allowed the instruction to gain persistence in the victim's ecosystem. As explained in the SociaLLM article, there is only one channel for instruction and data for LLMs, making this kind of attacks possible.
The attacker does not need access to the victim’s Microsoft 365 tenant. For the memory attack,the attacker only needs to control a web page that the victim asks Copilot to summarize.
The attack does not require code execution, malware, OAuth consent phishing, or credential theft.It exploits the way untrusted content is incorporated into Copilot’s context window and interpreted by the underlying LLM.
This is a great example of the role confusion technique: I do not know, nor do I need to know, Copilot’s exact internal message format for the class of attacks reported in this series to work. LLMs are tolerant parsers and they can therefore infer intent from approximate structure, such as pseudo-JSON.
During testing, I found that Copilot could be prompted to describe its perceived context and tool-call formats. Whether or not those descriptions were exact, they gave me useful candidate structures for constructing injection payloads.More importantly, the model did not require an exact match to an internal schema. It only needed to interpret the attacker-controlled content as authoritative enough to influence its response.
Part 2 is a more standard SociaLLM technique and leverages attacker-controlled incoming emails to inject malicious instructions.
Instructions embedded in attacker-controlled email messages could cause Copilot alter its behavior, presented as three variants:
Variant 1: Change the reported time of calendar events
Variant 2: Report that the organization is under cyber attack
Variant 3: Insert summaries of internal information into outbound email drafts addressed to an attacker\
The clever twist is to make the data exfiltration invisible to the unsuspecting person using Copilot as it appears outside the default viewport:
Copilot would search through either the victim’s inbox or OneDrive and then summarize what it found andinsert it below 50 newline characters in the email draft. To further hide the attack, the PoC hid this XPIA within a benign looking email.This had the added effect that Copilot would reply to the benign part of the email first, then add 50 newlines and insert the summary of the victim’s inbox or the OneDrive document, thereby making it seem like only the benign reply was generated.Because 50 newlines were added before the summary, the inserted summary was also placed outside the compose viewport.Thus, the victim had to both realize more content had been added after the initial benign reply draft and scroll down to see it.
The mitigation, as expected, is to use one of the design patterns to leverage a quarantined LLM for the summarization.
The last article is an example of LLM Worm, injecting instructions in an attacker-controlled document that action the attacker's intent AND copy the instruction to any other document the LLM manipulates in the process:
An attacker places hidden instructions in a document that is later used as source material in Copilot for Word. Copilot may interpret those instructions as part of the user’s request, causing it to manipulate the document being drafted or edited.Copilot may then also copy the hidden instructions into the resulting document, turning that document into a new carrier.If the carrier is subsequently used in another Copilot-assisted workflow, the instructions can trigger again and propagate into further documents, even without the attacker’s original document being present.
This one is particularly interesting as Microsoft did not find a way to mitigate it fully.
Unlike Parts 1 and 2, this scenario remains exploitable at publication. I have weighed that carefully. The coordination period agreed with Microsoft has been exhausted, and testing shows that no robust mitigation for the broader vulnerability class is currently available. [...]
I assume the dual-llm pattern doesn't work here, but I'm not sure if I understand why. Microsoft then had to fall back to classic guardrails which, as we know, are not foolproof by design. Deterministic guardrails will be bypassed by altering the payload:
Microsoft successfully mitigated the originally submitted PoC prompt, and deployed multiple fixes over the course of this disclosure. Each of these raised the bar by closing the specific payloads reported, and reproducing the behavior afterwards required altered payloads rather than reusing the old ones directly. [...]Using a modified payload, the complete attack chain has been reproduced with all mitigations deployed
The author also provides a very neat explanation as to why LLM-as-a-judge is not going to work either as it shares the same flaws as what it is trying to protect:
Because LLMs can recover semantics across radically different representations, an effective detector must possess comparable semantic recovery capabilities.A detector weaker than the target LLM will cover a smaller representational space, leaving malicious formulations that the target understands but the detector fails to recognize.
The only generally available technology with comparable semantic capabilities is another LLM.Placing one model in front of another may reduce the success rate of particular attacks, but it creates an “LLMs all the way down” problem, where every LLM introduced to protect another LLM must itself be protected.