A sweep of eight agent orchestration architectures across five instruction-tuned 7-9B models and six benchmarks found that scaling a team from three to thirty calls raises accuracy by up to 17 points …
TinyAIArena, an open-source project hosted on GitHub under the hp6/ai-arena repository, replaces static AI benchmarks with a turn-based visual match in which four models compete on a fixed 8x8 grid re…
Multiverse's new paper, "LLM Compression by Block Removal with Constrained Binary Optimization" (arXiv 2602.00161), reformulates block-removal pruning of large language models as a constrained binary …
TypeSafe released a new "System One" model called Jev that answers typed questions with a chosen answer, per-option probabilities, and a confidence score rather than generating text, priced at $0.042 …
Hamel Husain and Shreya Shankar published an AI Evals FAQ distilling the most common questions from teaching 700+ engineers and product managers about AI evaluation. The guide distinguishes model benc…
Researchers introduced QVAC Genesis III, a 191.43B-token STEM-focused synthetic corpus spanning 19 domains, built via a dual generation strategy that uses a weak edge-scale student model's failures as…
A training-free, CPU-only lexical prompt-compression pipeline reduced prompt tokens by a mean of 40.3% (sigma = 9.2) while holding BERTScore-F1 at 0.876 against original-prompt output, according to an…
Researchers introduced EAR, an Entity-Aware Partitioning approach for retrieval-augmented generation (RAG) that extracts normalized surface anchors from questions, answer options, and corpora to build…
Two MMLU accuracy scores of 0.781 for build 42 and 0.79 for build 44 of the acme-gpt-7b model family are structurally valid but return "incomparable" from the score-delta verifier in the apl-ai-eval c…
A developer outlined best practices for evaluating AI models, emphasizing a multi-layered framework combining standardized benchmarks like MMLU and HumanEval, adversarial red teaming for prompt inject…
A new study from arXiv (2609.02899v1) finds that benchmark contamination inflates absolute scores but rarely reorders large language model (LLM) leaderboards, with a rank correlation of 0.997 between …
LLM benchmark scores are becoming unreliable because models are trained on the same tests they are evaluated on, leading to inflated results that fail to reflect real-world performance, according to a…
A hands-on benchmark of Qwen3.8 27B on an RTX 4090 found that 4-bit quantization (NF4 and AWQ INT4) preserves near-baseline quality, with MMLU scores of 59.8% and 59.5% versus 61.2% for FP16, while 1-…
An independent researcher is seeking an arXiv endorsement for a paper titled “Diagnosing the Measurement Problem in AI Evaluation: Four Structural Failure Modes Across Six Benchmarks,” which analyzes …
A new arXiv study introduces the 'fragility grid,' showing that 12 open-weight instruction-tuned LLMs from 4 families score between 31 and 89 percent on the same 3,679 items from 4 benchmarks (ARC, He…
A new method called COPA reduces attack success rates by 6.3× versus the best static baseline and 4.4× on average across lifelong attack streams, while retaining 92% defense on month-old attacks compa…
Jie Tang's team at Z.ai argues that LLM scaling should optimize for fixed inference budgets, favoring smaller, deeper architectures trained longer over wider models. Their ablation shows a 7B model tr…
IBM Research has introduced BenchDrift, an open-source tool that quantifies how much large language model benchmark scores shift when test prompts are rephrased without changing meaning, revealing tha…
A new method called Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass, achieving 2.7–3.2x the throughput of standard fine-tuning at ~40% less peak training memo…
A developer's guide outlines the three pillars for building production-ready AI agents: benchmarking, cost optimization, and tooling. It emphasizes task-level evaluation over static benchmarks, using …