{"playbook": "evals-testing", "title": "Evals and testing for LLM apps", "description": "Measuring and regression-testing LLM apps and agents: eval datasets, benchmarks you run yourself, CI gates, tests on tool calls and traces.", "url": "https://wpnews.pro/playbooks/evals-testing", "count": 3, "recommendations": [{"advice": "When continuing harness optimization across a new benchmark or policy model, log each edit's rationale, predicted and observed outcomes, and eval context so later attempts can reuse or revise proven changes.", "takeaway": "Recording each harness edit's rationale, predicted and observed outcomes, and evaluation context in a Self-Improvement Memory lifted final scores by an average of 2.36 points over harness-only continuation across ten coding runs (p=0.003).", "article_url": "https://wpnews.pro/news/self-improvement-memory-enables-continual-recursive-self-improvement", "title": "Self-Improvement Memory Enables Continual Recursive Self-Improvement", "source_url": "https://arxiv.org/abs/2611.18856", "published_at": "2026-11-17T20:54:25+00:00"}, {"advice": "When prompting coding models for API calls, Dockerfiles or CI workflows, include a dated list of known upstream breaks with the fix for each; the eval cut broken-route picks from 37% to 16% for claude-sonnet-5.5, while a bare date did little.", "takeaway": "On 58 prompts where a dated upstream break stops code working, giving models an explicit list of dated breaks (date, title, fix) cut broken-route choices for claude-sonnet-5.5 from 37% to 16% and gpt-6.1-sol from 38% to 20%, while adding only today's date barely helped and the two GGUF open models g", "article_url": "https://wpnews.pro/news/dated-break-eval-do-coding-models-write-code-that-stops-working-on-a-known-date", "title": "Dated-break eval: do coding models write code that stops working on a known date?", "source_url": "https://discuss.huggingface.co/t/dated-break-eval-do-coding-models-write-code-that-stops-working-on-a-known-date/191992#post_1", "published_at": "2026-11-17T19:51:32+00:00"}, {"advice": "Check independent leaderboards like the Vals Index alongside vendor benchmarks before adopting a flagship model; GPT 6.2 Astra placed 7th there despite its launch.", "takeaway": "GPT 6.2 Astra ranks 7th on the Vals Index, behind both Gemini 4 Argon and Gemini 4 Hydrogen, per the independent leaderboard.", "article_url": "https://wpnews.pro/news/gpt-6-2-astra-places-7th-on-vals-index-behind-even-gemini-4-argon-gemini-4", "title": "GPT 6.2 Astra Places 7th On Vals Index, Behind Even Gemini 4 Argon & Gemini 4 Hydrogen", "source_url": "https://officechai.com/ai/gpt-6-2-astra-vals-index/", "published_at": "2026-11-17T19:40:14+00:00"}]}