A new study introducing BenchDrift shows that rephrasing benchmark problems while keeping meaning and answer fixed flips LLM correctness in both directions across eight models and three benchmarks (GS…
Researchers introduced LLM Agents Factory, a retrieval-based framework that constructs domain-specific LLM agents on demand from a base of over 20,000 predetermined agent profiles, achieving accuracy …
A new arXiv study (2608.09942v1) finds that chain-of-thought (CoT) prompting is not a universal enhancer of large language model reasoning, but rather a bandwidth bypass that helps only when serial co…
Computer Anthology, a new benchmark family from the ACL Anthology repository, evaluates AI models on reasoning over newly published NLP papers, with monthly updates to prevent data contamination. The …
Researchers introduced Gaokerena, a family of compact Persian medical language models, with Gaokerena-V improving a translated medical MMLU benchmark from 46.28% to 49.31% and Gaokerena-R achieving 52…
A new arXiv preprint (2607.28801v1) introduces a dataset-centric meta-evaluation framework that audits LLM benchmark datasets at the sample level across five dimensions, revealing internal heterogenei…
Researchers introduced TokenSwap, a method that replaces textual concepts with semantically aligned images to create image-interleaved inputs, and found that across 42 multimodal large language models…
A study submitted to arXiv on 30 Jul 2026 found that gently-compressed large language models (LLMs) can pass standard data-free quality guards—including perplexity, MMLU accuracy, and output-fidelity …
Redwood Research finds that hint-based chain-of-thought faithfulness evaluations still work on Claude models, contradicting Anthropic system card claims that recent models no longer use hints. The rep…
A study at 120B scale finds that inserting constitutional content during midtraining produces durable alignment gains, with constitutionally midtrained models outperforming a control on alignment gene…
A new open-source benchmarking framework, Inspect India Evals, reveals that large language models (LLMs) tested on Indian linguistic and cultural contexts show significant performance gaps, with Sarva…
Researchers present EaaS, a cloud-native reference architecture that operationalizes AI evaluation methods as six stateless Kubernetes microservices, achieving sub-2ms p99 latency for conformal predic…
Three prompting techniques that worked in 2023 now actively cost accuracy in 2026, according to recent research. Expert personas reduced factual accuracy to 68.0% versus 71.6% without on the MMLU know…
A new permutation diagnostic reveals that position bias in multiple-choice LLM benchmarks is only detectable within a 60-95% base-accuracy Goldilocks zone, with frontier-tier models operating above th…
A new study from arXiv introduces an uncertainty-aware trust estimation method for multi-LLM systems, adapting Cooke-style log weighting from structured expert judgment to aggregate predictions from h…
A new benchmark and training set called SelectBench, introduced by researchers in arXiv:2607.20090v1, shows that reinforcement learning via DAPO can modestly improve selective evidence adoption in ret…
A new arXiv preprint (2607.16799v1) finds that hidden-state readouts from language models primarily encode self-judgement (SJ) rather than objective correctness (OC), with the SJ-associated direction …
A developer's team replaced subjective 'looks good to me' testing with an automated LLM evaluation pipeline that catches 92% of hallucinations before deployment. The system uses a judge ensemble inclu…
A benchmark score is a manufactured number that passes through a chain of choices—sampling, prompting, scoring, and aggregation—each of which can change the final result while model weights stay fixed…
Researchers at arXiv introduce an evaluation-unsupervised benchmark coreset selection method that uses submodular subset selection to pick a small subset of prompts from multiple LLM benchmarks while …