Fortitude Omnis Group's OmnisBench benchmark initially showed small models performing surprisingly well, but community feedback revealed potential contamination from old benchmarks like HumanEval and …
Researchers introduced Adversarial Review (AR), a minimal cooperative code-review protocol with a main coding agent, a reviewer, and a critic, achieving the highest pass rate on LiveCodeBench among te…
A controlled experiment by researchers testing Claude Opus 4.7 and Codex GPT-5.5 on 116 Python tasks found that pairing a weaker reviewer with a stronger writer can degrade performance: Claude's 91.4%…
A new arXiv preprint (2608.00355v1) finds that apparent progress toward harder tasks in large language models is mostly a ceiling effect, but a smaller hard-task effect persists after controlling for …
A new arXiv paper (2607.17531v1) introduces a fixed-pool diagnostic framework for test-time collaboration in LLMs, decomposing net gains into recoverable mass, verification-signal coverage, conditiona…
Researchers introduced DecompRL, a reinforcement learning algorithm that breaks complex problems into smaller sub-tasks for large language models, reducing GPU token costs by approximately 50 times. T…
Researchers at an undisclosed lab introduced a best-of-N (BoN) optimization method as a proxy for self-play training in debate protocols, aiming to improve scalable oversight for AI systems. Their exp…
Google's Gemini 2.5 Pro with Deep Think reasoning mode topped coding and reasoning benchmarks this week, scoring 82.4% on GPQA Diamond and 94.1% on HumanEval+, but the mode multiplies token costs by r…
Engineers can now run a full local agentic programming stack using Ollama, Gemma 4, and Claude Code, bypassing API costs, rate limits, and data privacy concerns. Google DeepMind's Gemma 4 26B MoE mode…
Researchers have developed BenchEvolver, a system that automatically generates harder coding benchmarks by first evolving solution algorithms and then deriving new problem statements from them. The me…
Researchers have introduced BOHM, a zero-cost attribution method for compound AI systems that extracts hierarchical attribution trees directly from existing routing weights. The method provides multi-…