Researchers introduced DeltaML-Bench, a benchmark of 48 tasks from research papers that tests AI agents' ability to improve published baselines in real-world repositories. In evaluations, the ARG scaf…
Researchers introduced ClosureBench, a constructive benchmark for compositional graph-relational reasoning that generates instances on demand with programmatically verified ground truth, avoiding data…
In a benchmark of 10 leading AI models reviewing enterprise test case documentation, 9 out of 10 fell into 'Abstract Complacency,' merely summarizing headings without verifying claims. Only Kimi K3 ac…
Researchers introduced SKILL, a self-correcting knowledge-guided iterative large language model agent that unifies multi-agent LLM reasoning with reinforcement learning for logic synthesis optimizatio…
A developer built a free browser-based token counter that handles OpenAI, Anthropic, and Google models, providing exact or estimated token counts and custom pricing. The tool aims to help developers a…
AWS Bedrock provides up to three times more notice than Anthropic before Claude models are deprecated, according to an analysis of 19 model retirements. Anthropic's median notice was 63 days, while Be…
AI Change Watch, an independent project tracking model deprecations, found that Anthropic gives at least 60 days' notice before retiring a model, with a median of 63 days across 19 retirements, while …
A new study from the Supervised Program for Alignment Research (SPAR), led by Achu Menon and mentored by Santiago Aranguri of Goodfire, finds that the realism win rate, a metric used to measure evalua…
A new guide explains how to implement the LLM-as-judge pattern to automatically evaluate the quality of AI responses in production, addressing the challenge of manually reviewing high volumes of outpu…
A new guide details the LLM-as-judge pattern for automatically evaluating AI response quality in production, addressing the challenge of monitoring high-volume outputs. The approach uses a more powerf…
A new guide explains how to implement the LLM-as-judge pattern to automatically evaluate AI response quality in production, addressing the challenge of scaling quality assurance for high-volume AI sys…
ChatPlayground AI is offering a lifetime Unlimited Plan license for $59.25 with code Aug25, down from $619, until August 9 at 11:59 p.m. The platform runs one prompt across more than 20 models simulta…
A developer's colleague incurred a $1,200 bill in a two-hour AI coding session, highlighting the hidden costs of AI assistants. Token consumption, especially for code, is 1.5-2.5x higher than prose, a…
A two-year analysis by Soamee, an AI integration firm, reveals that the real cost of running large language models in production is driven by inflated input token counts from system prompts, conversat…
A test of five modern LLMs on undergraduate music theory found that GPT 5.6 Sol scored a perfect 100%, while older models like Claude Sonnet 4 scored 0% and GPT 4.1 scored 16%, indicating LLMs have su…
Hallmark, an open-source design skill by Hassan El Mghari (Nutlope) at Together AI, has crossed 12,300 GitHub stars by preventing AI coding agents from generating homogeneous landing pages. The skill …
Soamee, a company building AI features for clients, reports that the real cost of running LLMs in production is often underestimated due to system prompts, conversation history, and RAG context inflat…
Soamee, a company that develops AI features for clients, reports that the real cost of operating large language models (LLMs) in production is often underestimated, with system prompts, conversation h…
A developer who built CourtGPT.ai to serve 6.7M+ legal records shares 18 months of lessons on production RAG systems. Key findings include that hybrid search (BM25 + vector) achieves 92% recall at top…
A new closed-loop AutoML framework using GPT-5, GPT-4o, and Claude Sonnet 4 as autonomous neural architecture designers achieves mean test accuracies above 93% and a best accuracy of 98.1% for cross-l…