HumanEval

Researchers have introduced DeepSuite, a new benchmark for evaluating AI coding agents that tests them using short, natural-language prompts requiring complex solutions across entire code repositories…

A developer argues that current LLM benchmarks like MMLU, HumanEval, and SWE-bench measure only knowledge recall and one-shot task completion, not the behavioral traits—such as debugging, adaptation, …

The Gemma 4 26B model was the first local AI to achieve a perfect score on the HumanEval benchmark, including solving the notoriously difficult problem 145. This problem requires sorting integers by t…