cd /news/ai-safety/when-objectives-and-evaluators-disag… · home › topics › ai-safety › article
[ARTICLE · art-173021] src=arxiv.org ↗ pub= topic=ai-safety verified=true sentiment=· neutral

When Objectives and Evaluators Disagree: Evaluating Transfer for Post-hoc Demographic Framing Mitigation

A study posted to arXiv (2611.25258v1) finds that post-hoc rewriting policies for reducing demographic framing disparities in LLM-generated text transfer inconsistently across objectives, evaluators and prompts, tested on 660 gender pairs and 1,980 age contrasts of demographic-conditioned climate messages. An adaptive policy that proposes edits on both sides but accepts at most one beat a fixed lower-Persuasion Bias Index targeting rule in seven of eight comparisons across two evaluator families, mostly by editing more pairs rather than by guided direction choice, and at higher preservation risk. The authors conclude mitigation should be reported as transfer across objectives, evaluators and editing policies, noting that shrinking the PBI gap increases dimension-mean disparity under both evaluator families.

recommendation

Report demographic-framing mitigation as transfer across objectives, evaluators and prompts, and validate rewrites with an independent evaluator rather than the rewriter's own selector.

more in playbook: Evals and testing for LLM apps
by read1 min views1 publishedNov 20, 2026

arXiv:2611.25258v1 Announce Type: new Abstract: Post-hoc rewriting is used to reduce demographic disparities in LLM-generated text when the source model is inaccessible. A rewrite can look successful merely because it improved its own target or satisfied its selector. We study this as a transfer problem on 660 gender pairs and 1,980 age contrasts of demographic-conditioned climate messages. A rewriter proposes edits, a selection model chooses among constraint-satisfying candidates, and an independent evaluator uninvolved in either rates pairwise framing disparity. A fixed policy that always targets the message with the lower Persuasion Bias Index (PBI) transfers inconsistently across axes and outcome aggregations. An adaptive policy that proposes edits on both sides but accepts at most one significantly improves on that rule, applied to the same candidates, in seven of eight comparisons across two evaluator families, mostly by editing more pairs, not by guided direction choice, at higher preservation risk. A policy editing both messages gives the largest point reductions on fewer pairs. These two policies reduce disparity across a preservation-threshold sweep; absolute ratings vary across evaluator families and prompts. Shrinking the PBI gap increases dimension-mean disparity under both evaluator families. Mitigation should be reported as transfer across objectives, evaluators and their prompts, and editing policies.

── more in #ai-safety 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/when-objectives-and-…] indexed:0 read:1min 2026-11-20 · —