cd /news/large-language-models/raw-refusal-aversion-in-the-wild-a-c… · home › topics › large-language-models › article
[ARTICLE · art-171583] src=arxiv.org ↗ pub= topic=large-language-models verified=true sentiment=· neutral

RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs

A new arXiv paper (2611.21641v1) introduces RAW, a method that uses LLM stochasticity as a natural experiment to causally measure the user-side cost of refusals in deployed models from logs alone. Applied to 15 deployed LLMs in LMSYS-Chat-1M and WildChat with user-clustered inference, RAW found that for about half of the precisely measurable models a refusal costs 3 to 17 percentage points of subsequent re-engagement, while the rest show no distinguishable effect — including one deployment whose positive point estimate traced to persistent users retrying the refused request. The authors release labels, moderation scores, prompt embeddings and pipeline code, and report the method runs on any deployed model for about $0.30 per 1,000 conversations.

recommendation

Measure refusal's user-side cost by comparing near-identical prompts where the same model sometimes refuses and sometimes complies, then cluster inference by user to avoid mistaking retries for retention.

more in playbook: Evals and testing for LLM apps
by read1 min views4 publishedNov 18, 2026

arXiv:2611.21641v1 Announce Type: new Abstract: Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API. We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal. Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request. Our contribution is the measurement method, which runs on any deployed model from logs alone for about $0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.

── more in #large-language-models 4 stories · sorted by recency
── more on @raw 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/raw-refusal-aversion…] indexed:0 read:1min 2026-11-18 · —