{"slug": "optimismbench-forecasting-bias-and-the-alignment-effect-in-language-model", "title": "OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment", "summary": "A new study from arXiv introduces OptimismBench, a benchmark that detects directional bias in large language models' probability judgments by using inverted pairs to measure asymmetry between P(success) and P(failure) without ground truth. Testing 16 models from 8 providers, the study found that 14 are optimistic, with pessimism only in Anthropic's frontier tier, and that post-training sets the bias sign across 11 matched base-versus-chat pairs. The pattern persists across ablations, and model identity dominates language, with inter-model variance at 4.7x inter-language variance in a 17-model, 6-language comparison.", "body_md": "arXiv:2607.26981v1 Announce Type: new\nAbstract: Large language models are increasingly used as decision aids whose probability judgments shape downstream choices. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground-truth probability. When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags. We introduce OptimismBench, which detects directional bias with inverted pairs: each scenario elicits both P(success) and P(failure), and asymmetry between the two framings yields a signed bias score without ground truth. Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic's frontier tier. Eleven matched base-versus-chat pairs across four families show post-training sets the sign of the bias, with opposite shifts in different families. The pattern survives prompt, temperature, perspective, and self-debiasing ablations. A seventeen-model six-language comparison further shows model identity dominates language, with inter-model variance at 4.7x inter-language variance. We release 3,870 items across 10 languages for per-model directional-bias auditing. When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default.", "url": "https://wpnews.pro/news/optimismbench-forecasting-bias-and-the-alignment-effect-in-language-model", "canonical_source": "https://www.machinebrief.com/news/optimismbench-forecasting-bias-and-the-alignment-effect-in-l-novp", "published_at": "2026-07-30 04:00:00+00:00", "updated_at": "2026-07-30 05:34:16.531128+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-ethics", "ai-research"], "entities": ["arXiv", "OptimismBench", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/optimismbench-forecasting-bias-and-the-alignment-effect-in-language-model", "markdown": "https://wpnews.pro/news/optimismbench-forecasting-bias-and-the-alignment-effect-in-language-model.md", "text": "https://wpnews.pro/news/optimismbench-forecasting-bias-and-the-alignment-effect-in-language-model.txt", "jsonld": "https://wpnews.pro/news/optimismbench-forecasting-bias-and-the-alignment-effect-in-language-model.jsonld"}}