A new benchmark and training set called SelectBench, introduced by researchers in arXiv:2607.20090v1, shows that reinforcement learning via DAPO can modestly improve selective evidence adoption in ret…
A new arXiv preprint (2607.16799v1) finds that hidden-state readouts from language models primarily encode self-judgement (SJ) rather than objective correctness (OC), with the SJ-associated direction …
A developer's team replaced subjective 'looks good to me' testing with an automated LLM evaluation pipeline that catches 92% of hallucinations before deployment. The system uses a judge ensemble inclu…
A benchmark score is a manufactured number that passes through a chain of choices—sampling, prompting, scoring, and aggregation—each of which can change the final result while model weights stay fixed…
Researchers at arXiv introduce an evaluation-unsupervised benchmark coreset selection method that uses submodular subset selection to pick a small subset of prompts from multiple LLM benchmarks while …
Researchers have developed a new approach to overcome the quadratic cost of causal self-attention in long-context transformers, using state update design and structural interventions like sink tokens …
A PlainEnglish article warns that AI benchmark scores such as MMLU, HumanEval, and HellaSwag can overstate production readiness when leaderboard numbers are treated as proof of model quality. The comm…
A new study finds that prompt robustness in large language models depends on the type of question asked, with objective questions showing different sensitivity to prompt variations than subjective bel…
Researchers introduce PuzzleMoE, a method for compressing large Mixture-of-Experts models via fine-grained element-wise merging and bit-packing, achieving up to 16.7% higher accuracy on MMLU at 50% co…
The Center for AI Safety, with world experts, created Humanity's Last Exam (HLE), a benchmark of over 2,500 expert-level questions across disciplines to test AI reasoning. Even top models like GPT, Ge…
A reinforcement learning-driven data scheduler, AC-ODM, boosts MMLU performance by 27.5% relative and HumanEval pass@1 by 2.23× on a Pythia-1B model with only a 0.4% per-step wall-clock increase and 2…
A developer recounts pushing an agent into production that failed after a CRM dropdown change, highlighting the inadequacy of model-level evaluation for agent systems. The article argues that agents m…
Hugging Face and the EvalEval Coalition launched an integration that allows contributors to submit standardized evaluation results (EEE schema) to Hugging Face Community Evals, consolidating scattered…
Nearly half of 60 popular AI benchmarks studied in a new ICML paper show high levels of saturation, meaning they can no longer reliably distinguish between leading models. The researchers from the pap…
Recent AI research introduces RL-driven agentic optimization using dense token-level supervision and progress advantage signals to stabilize training. PhysiFormer injects 3D geometric reasoning into d…
A viral chart predicting open-weights AI models will catch closed frontier models by December 2026 is misleading, as analysis across 18 benchmarks shows the gap varies by task and is not uniformly shr…
A developer observed that after multiple context compactions in LLM sessions, output quality degrades non-linearly, with a brief improvement after the second compaction before declining. They built a …
Researchers proposed Dynamic-dLLM, a training-free framework to accelerate Diffusion Large Language Models (dLLMs) by dynamically allocating cache budgets and calibrating decoding thresholds. The meth…
A practical guide for benchmarking fine-tuned models recommends starting with a held-out test set matching the actual task rather than relying solely on public benchmarks. The workflow includes defini…
Microsoft Principal Developer Advocate Waldek Mastykarz published a blog post arguing that testing large language models in empty chat sessions is methodologically flawed, as models have no preference…