cd /news/ai-safety/fifty-four-percent-down-to-point-two… · home › topics › ai-safety › article
[ARTICLE · art-174041] src=gautamparab.com ↗ pub= topic=ai-safety verified=true sentiment=· neutral

Fifty-Four Percent Down to Point Two, Scored by the House

OpenAI reported on 20 November that the average attack success rate for extracting a model's protected chain of thought fell from 54.0% against GPT-6 Astra with its original safeguards to 0.2% against GPT-6.2 Astra with updated safeguards, a factor of 270, and from 22.9% to 0.9% for current-request attacks. The figures come from OpenAI's own in-house "strong automated red-teaming" aggregated across six unnamed attacker models, with no external measurement, and the drop bundles four mitigations landed at once: adversarial training against extraction attempts, input and output classifiers that inspect tool-call arguments, and API rules tying reasoning to the originating conversation across user, workspace, organization and model-family boundaries. The post cites Ma and colleagues' 2024 Machine Learning paper, which found 11 of 24 adversarial-training defenses published since 2018 lost more than a percentage point of previously measured robustness under stronger attacks, and Apruzzese and colleagues' IEEE SaTML 2023 survey of 88 adversarial-ML papers from 2019-2021, of which 20% ran experiments on real ML systems and 51% released source code.

recommendation

Evaluate reasoning-extraction defenses against independent, stronger red-team attacks rather than in-house ones alone, since reported robustness has repeatedly fallen by up to two orders of magnitude when the attacker improves.

more in playbook: Evals and testing for LLM apps
read6 min views1 publishedNov 22, 2026
Fifty-Four Percent Down to Point Two, Scored by the House
Image: Gautamparab (auto-discovered)

On 20 November OpenAI published a post on what it calls adversarial distillation: someone using a frontier model’s outputs, without authorization, to train a cheaper model that inherits the capabilities and not the safeguards. Most of the post is description of defenses. One table carries the claim.

The table reports attack success rate, or ASR, for attempts to pull a model’s protected reasoning, its chain of thought, back out of it. Against GPT‑6 Astra with its original safeguards, attacks that targeted reasoning from an earlier request succeeded 54.0% of the time. Against GPT‑6.2 Astra with the updated safeguards, 0.2%. For the current request the pair is 22.9% and 0.9%. The prose version: “The average attack success rate fell from 54.0% against GPT‑6 Astra with its original safeguards to 0.2% against GPT‑6.2 Astra with its updated safeguards.”

A 54 to 0.2 drop is a factor of 270. I want to be careful about what it is a measurement of, because ASR is not a property of a model. It is a property of a model and an attacker, jointly. The defender half of that pair got rebuilt between the two columns. The attacker half is, by OpenAI’s own description, “strong automated red-teaming” run in-house, aggregated across six attacker models it does not name. Nobody outside the company produced either number.

The number that moves when someone tries harder #

Adversarial robustness research spent a decade discovering that defenses reported as strong had been measured against attacks too weak to find the hole.

Ma and colleagues, in Machine Learning in 2024, describe one mechanism they call imbalanced gradients: one term of the loss dominates and steers the attack somewhere suboptimal, so the defense looks robust when it is not. They build a margin decomposition attack to exploit it and run it against 24 adversarial-training defenses published since 2018. Eleven of the 24 lose more than a percentage point of their previously best-measured robustness.

The sharpest case in that paper is not a defense at all, just a training trick. Add label smoothing to a naturally trained model and the standard PGD attack measures 7.87% robustness. The stronger attack finds 0.06%. Same model, same threat model, same metric: two numbers 131-fold apart, and the only thing that changed was who was attacking. On ImageNet their attack alone beats the AutoAttack ensemble that RobustBench reports, scoring XCiT-S at 40.26% where AutoAttack had 41.78%.

Reported robustness gets revised in one direction, downward, as attacks improve, and the size of the revision ranges from fractions of a point to two orders of magnitude depending on whether the original evaluation happened to match the weakness. OpenAI’s 0.2% is a claim of the same type, and the only control on it is that the same team also produced the 54.0%. If their red-teamers are strong, the pair is meaningful. If their red-teamers are weak, both numbers are too low and the ratio between them can still look terrific.

The company does say what it did to earn the drop: adversarial training against extraction attempts, input and output classifiers that also inspect tool-call arguments, and API rules tying reasoning to the conversation that produced it, including across user, workspace, organization and model-family boundaries. Those are specific, plausible changes. They are also the reason the drop is hard to read: four mitigations landed at once, and the table scores the bundle.

What a checkable version would look like #

The gap between an internally measured defense and an independently measured one is the subject of a position paper I keep returning to. Apruzzese and colleagues, at IEEE SaTML in 2023, surveyed every adversarial-ML paper at CCS, USENIX Security, NDSS and IEEE S&P from 2019 through 2021, 88 papers in all, and counted what they actually tested on. Twenty percent ran experiments on real ML systems. Fifty-one percent released their source code. Their title is the practitioner’s complaint: “Real Attackers Don’t Compute Gradients.”

Their recommendations to companies are modest, and they are roughly the list I would hand OpenAI here: run bug-bounty programs with clear interaction guidelines, dedicate staff to processing researcher requests, publish high-level schematics of the system so outsiders can design realistic experiments. The argument is not that companies should open-source their production stack. It is that a security claim about a closed system needs a route by which somebody outside can try to break it and report what happened.

OpenAI has part of that route already, which is why I think the September disclosure is the stronger document. It credits independent security researchers for bringing cross-model and conversation-compaction attack paths through responsible disclosure, and says the company investigated and “confirmed that the attack paths they identified were real.” That sentence is the only load-bearing finding in either post that originated outside the company. It is also not a rate. It is a yes.

The enforcement numbers have no denominators either #

The November post says weekly enforcement against distillation campaigns has “ranged in scale from thousands to tens of thousands of accounts, with the rate increasing significantly this past summer.” September is more specific. Activity began on 1 July at low volume; spikes on 24 and 25 July amounted to 16,000 requests using a relevant extraction pattern from over 4,000 users; investigation found related prompt-pattern activity across a cluster of more than 15,000 users, disrupted by 28 July. OpenAI attributes a core cluster to individuals associated with Moonshot AI, the developer of Kimi, the same lab

that showed up in Anthropic’s threat-intelligence report in September. The footnote on those figures is the most honest line in either post: “These figures describe attempted, not necessarily successful, extractions.” So the record gives a count of attempts and a rate of success, measured separately, one by production logging and one by an internal red team, and no way to multiply them. Sixteen thousand attempts at an unstated success rate, against a defense whose success rate is reported for a different set of attempts entirely.

What I would want before believing 0.2% #

Three things, none of which require giving away anything an attacker could use. How many attempts does the 0.2% divide? One failure in 500, or one in 5,000? What budget did each attacker model get, in turns and tokens, since an adaptive attacker that gives up early produces a lovely number? And did anyone outside the building replay the attack suite against the shipped system?

The September post shows the company can answer that third question in the affirmative when it wants to, because outside researchers found real attack paths and it said so. The November post, which is the one making a quantitative claim, does not. Until it does, 54.0 to 0.2 is a statement about the relationship between two versions of a model and one red team, which is worth something, and not a statement about what an adversary can get, which is what it will be read as.

References

  1. OpenAI, “Combatting adversarial distillation,” 20 November 2026. openai.com/index/combatting-adversarial-distillation
  2. OpenAI, “Disrupting a coordinated model-distillation campaign,” 30 September 2026. openai.com/index/disrupting-a-coordinated-model-distillation-campaign
  3. Ma X, Jiang L, Huang H, Weng T-W, Bailey J, Jiang Y-G. “Imbalanced gradients: a subtle cause of overestimated adversarial robustness.” Machine Learning , May 2024.doi:10.1007/s10994-023-06328-7
  4. Apruzzese G, Anderson HS, Dambra S, Freeman D, Pierazzi F, Roundy K. “‘Real Attackers Don’t Compute Gradients’: Bridging the Gap Between Adversarial ML Research and Practice.” 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , February 2023.doi:10.1109/SaTML54575.2023.00031
── more in #ai-safety 4 stories · sorted by recency
── more on @openai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/fifty-four-percent-d…] indexed:0 read:6min 2026-11-22 · —