The Allen Institute for AI introduced BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts, which uses multidimensional Item Response Theory to separate the underlyin…
A developer argues that AI agents' failures in production are due to memory and state management issues, not reasoning limitations. The post outlines a multi-layered memory architecture—short-term, lo…
Multigrid AI explains how graduate-level science benchmarks like GPQA are constructed, emphasizing the use of a non-expert baseline to make scores interpretable. The process involves domain experts wr…
Redwood Research finds that hint-based chain-of-thought faithfulness evaluations still work on Claude models, contradicting Anthropic system card claims that recent models no longer use hints. The rep…
Model benchmarks are creating a fragmented AI landscape where 'the best model' depends on which specific test is valued most, according to a news analysis. GPT-5.6 leads on GPQA while Opus 5 dominates…
Researchers introduce Depth-Entropy Guided Sampling (DEGS), a training-free test-time method that exploits layer-wise entropy collapse in transformer forward passes to improve LLM reasoning. Across th…
Researchers introduced LearnStop, a cost-aware early-exit method for reasoning language models that learns when to stop computation based on online features like answer confidence and entropy. Across …
Anthropic released Claude 3.5 Sonnet, a new AI model that outperforms the previous top-tier Claude 3 Opus in intelligence, speed, and cost. The model achieves a 64% solve rate on internal agentic codi…
A new study from researchers testing adapter composition in large language models found that geometry-aware merging strategies, which enforce orthogonality or directional independence in parameter upd…
NVIDIA researchers developed a task-seeded synthetic Q&A generation workflow for Nemotron-family pretraining that uses public task training splits as capability seeds to generate new task-aligned exam…