A team led by Ming Zhong at the University of Illinois Urbana-Champaign and Google DeepMind has introduced SWE-IF, a framework that aligns code evaluation with human preference, revealing that instruc…
Fortitude Omnis Group's OmnisBench benchmark initially showed small models performing surprisingly well, but community feedback revealed potential contamination from old benchmarks like HumanEval and …
Jie Tang's team at Z.ai argues that LLM scaling should optimize for fixed inference budgets, favoring smaller, deeper architectures trained longer over wider models. Their ablation shows a 7B model tr…
A new measurement study across four models finds that the default LoRA target_modules of q_proj and v_proj underperforms a data-driven placement for capability retention: on Llama-3-8B fine-tuning cod…
A new empirical analysis from arXiv (2608.19072v1) finds that large language model (LLM) agents post-training an LLM lock in their training strategy at the very beginning and spend the remaining budge…
The developers behind the open-source OmnisBench project have released a benchmark designed to measure the true cost savings of LLM routing. Running 364 coding and math tasks across four models, they …
A vendor-agnostic framework from DORA and other sources shows that AI coding tools should be judged by delivery outcomes such as change failure rate and time to restore service, not by vendor benchmar…
A developer's guide outlines the three pillars for building production-ready AI agents: benchmarking, cost optimization, and tooling. It emphasizes task-level evaluation over static benchmarks, using …
A new arXiv preprint (2608.12599v1) introduces a method to measure and restore constraint influence in black-box LLM dialogues, finding that behavioral relapse—models continuing to enact revoked const…
Researchers propose CORA-Diff, a training-free method that uses native confidence and persistence signals to skip unnecessary denoising steps in diffusion language models, achieving up to 3.32x speedu…
A developer built a 30-minute harness that tests coding models against five tasks mined from their own git history, scoring results with their real test suite. The harness, which runs any coding model…
A new arXiv paper (arXiv:2608.10333v1) introduces MERA, a method that improves small language models through multi-cycle adaptation, raising Qwen2.5-Coder-1.5B from 28.7% to 49.7% pass on held-out Hum…
An agent that passes 75 out of 100 test runs has a 75% pass@1 rate, but reliability depends on the product: pass@k (at least one success in k tries) yields 98.4% for k=3, while pass^k (all k succeed) …
A new tutorial from Anthropic demonstrates that AI agents can reliably self-correct only when grounded in external verification, citing a 2024 paper showing large language models cannot self-correct r…
DeepSeek V3 generally outperforms Claude 3.5 Sonnet on raw coding benchmarks like HumanEval and MBPP, especially in C++, Rust, and mathematical implementations, while Claude 3.5 Sonnet is considered s…
A developer argues that LLM evaluations for developer tools must measure three distinct axes: usefulness, correctness, and safety. Unlike chatbot evals, dev-tool evals rely on ground-truth signals lik…
A new study reveals that the multiplicative repetition penalty used across major LLM inference engines (HuggingFace, vLLM, llama.cpp) is fundamentally flawed because it branches on the sign of raw log…
XAI's Grok 4.5 scored 83% on Terminal Bench, outperforming many costlier models in multi-step coding tasks, while Anthropic's Claude Opus 4.8 excels in complex reasoning and high-stakes professional w…
A PlainEnglish article warns that AI benchmark scores such as MMLU, HumanEval, and HellaSwag can overstate production readiness when leaderboard numbers are treated as proof of model quality. The comm…
NVIDIA open-sourced Nemotron-Labs-TwoTower, a diffusion language model that generates text 2.42x faster than its autoregressive counterpart without retraining original weights. The model achieves 98.7…