{"slug": "lai-144-your-eval-improved-did-your-ai", "title": "LAI #144: Your Eval Improved. Did Your AI?", "summary": "Towards AI co-founder and head of community Louis-François Bouchard advised in newsletter LAI #144 that AI teams change the generation prompt and the judge prompt in separate evaluation runs, because updating both at once makes an eval score increase unattributable to either the answers or the scoring. Bouchard recommended recording each run and, when both prompts must change, running all four prompt combinations with the model and evaluation examples held fixed, so a gain that appears only under the new judge is read as an evaluation change rather than a product improvement. The issue also highlighted Research Operating System, a v1.0.0 open-source set of reusable protocols by community member Fadlih6481 for verifying evidence, citations, research gaps and methodology in AI-generated research, and was sponsored by Adobe for Claude, which brings 80+ creative and productivity tools including Acrobat into one Claude plugin.", "body_md": "Good morning, AI enthusiasts!\n\nYour eval score went up. That does not necessarily mean your AI got better.\n\nIf you change the generation prompt and the judge prompt in the same run, you no longer know what caused the improvement. The generated answers may be better, or your new judge may simply prefer them. This week’s AI tip shows a simple way to separate the two and keep your evals useful.\n\nWe also have some great technical reads this week. You’ll learn:\n\nPlus, an open-source research workflow from the community for checking evidence, citations, research gaps, and methodology instead of simply trusting generated research.\n\nLet’s get into it.\n\nThis week in What’s AI, I’m writing about something I’ve been noticing on our own team: the people delegating the most technical work to AI, especially AI engineers, sometimes produce weaker work outside the code itself.\n\nThe code can be great. But the GitHub description, documentation, explanation, or final presentation gets less attention than it used to.\n\nI don’t think the problem is simply “AI makes us worse at writing.” In the article, I look at a few reasons this might be happening, including how easy it is to accept polished output without giving it the same scrutiny we would give our own work. I also share what our team and I are doing differently to keep the final judgment and edit with the person responsible for the work.\n\nWhenever we set up evaluations, we change the evaluator and the system component being tested separately. For example, in our [Full Stack AI Engineering](https://towardsai.com/academy/full-stack-ai-engineering/?utm_source=newsletter&utm_medium=email&utm_id=AItips) course, we test changes to the generation prompt without changing the judge prompt at the same time.\n\nThe generation prompt changes the answers being produced. The judge prompt changes how those answers are scored. If you update both in the same run and the score increases, you lose that attribution. The answers may have improved, or the new judge may simply be scoring them more favorably.\n\nFor every eval run, record:\n\nIf you need to change both prompts, run all four combinations:\n\nKeep the model and evaluation examples fixed.\n\nIf the new generation prompt performs better under both judges, you have stronger evidence that the generation improved. If the gain appears only with the new judge, the evaluation changed, not necessarily the product.\n\n*— Louis-François Bouchard, Towards AI Co-founder & Head of Community*\n\nThis issue is brought to you thanks to [Adobe](https://www.adobe.com/go/adobe-for-claude):\n\n**Adobe for Claude** now brings 80+ pro-grade creative and productivity tools across imaging, design, video, and documents.\n\nFor the first time, Acrobat tools join Adobe’s creative tools in one Adobe plugin. And now you can go beyond prompting: new interactive editors let you work hands-on with PDFs and Adobe Express designs directly inside Claude, editing text, marking up documents, or fine-tuning individual design elements without leaving the conversation.\n\nStart creating and getting more done with the new Adobe for Claude plugin today.\n\n**Download the new plugin in Claude here***.*\n\n[Fadlih6481](https://discord.com/channels/702624558536065165/983037843532308500/1551393317722525736) built Research Operating System, an open-source set of reusable protocols for doing research with tools like ChatGPT, Claude, and Gemini. It is designed for cases where an AI-generated research output looks convincing, but the evidence, citations, research gap, or methodology behind it may not hold up. The protocols add more structure to tasks such as literature reviews, identifying research gaps, checking evidence, designing methodology, and planning experiments, with an emphasis on verifying the work rather than accepting the generated answer. [Check out the GitHub repo](https://github.com/afadlih/research-operating-system) and support a fellow community member. The project is still at v1.0.0, [so your feedback might help the creator](https://discord.com/channels/702624558536065165/983037843532308500/1551393317722525736).\n\nThe Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, [join the collaboration channel](https://discord.gg/rj6m9AF7eC)! Keep an eye on this section, too — we share cool opportunities every week!\n\n1. [Manish05817](https://discord.com/channels/702624558536065165/1548966009845719070/1548966009845719070) is looking for an AI learning partner to build something interesting together. It can be projects, experiments, hackathons, or even exploring new ideas and technologies. If this sounds like something you would like to do, [connect with him in the thread](https://discord.com/channels/702624558536065165/1548966009845719070/1548966009845719070)!\n\n2. [Jay1900707](https://discord.com/channels/702624558536065165/1548739611000578111/1548739611000578111) is looking for a collaborator to build an agentic AI project. They want to build something functional, iterate fast, and turn it into an open-source repo or working tool. If you are comfortable with Python and want to get into the agentic space, [reach out to them in the thread](https://discord.com/channels/702624558536065165/1548739611000578111/1548739611000578111)!\n\n3. [Busycat1](https://discord.com/channels/702624558536065165/1549376004542431343/1549376004542431343) is looking for a few collaborators to research and build an AI guardrails platform. If you are interested, [contact them in the thread](https://discord.com/channels/702624558536065165/1549376004542431343/1549376004542431343)!\n\nMeme shared by [bin4ry_d3struct0r](https://discord.com/channels/702624558536065165/830572933197201459/1552558881513672728)\n\n[Working Around a Model’s “No”](https://pub.towardsai.net/working-around-a-models-no-c56089f52d03?sk=1d900f1386f919e8fddd22e7974ccad5) By [Enzo Lombardi](https://enzolombardi.net/?source=post_page---byline--c56089f52d03-----------------------------------------)\n\nResearch has shown that refusal behavior in some language models can concentrate along a small number of directions in the residual stream. This article explains abliteration, which estimates a refusal direction from contrasting prompts and then projects it out of model weights to reduce refusals. It also covers the GGUF tensor format and the open-source tooling used to modify and distribute uncensored model variants, showing how much safety behavior can depend on relatively localized model representations.\n\n1. [When Can Old LLM Reasoning Traces Still Be Reused?](https://pub.towardsai.net/when-can-old-llm-reasoning-traces-still-be-reused-149e890adb2d?sk=cd6bc881005bae0aac75b477cb8b6e13) By [Shenggang Li](https://medium.com/@datalev?source=post_page---byline--149e890adb2d-----------------------------------------)\n\nReasoning traces collected from older checkpoints become less reliable as the policy changes during training. This article uses GRPO and off-policy evaluation to show how small token-level differences compound into large trajectory-level importance weights, reducing the effective sample size of historical data. Across GSM8K and SVAMP experiments, refreshing older traces improves evaluation accuracy, while simple policy-overlap thresholds do not reliably tell you when a trace is safe to reuse.\n\n2. [The sys.exit(0) Exploit: How AI Agents Fake Success](https://pub.towardsai.net/the-sys-exit-0-exploit-how-ai-agents-fake-success-736b2fb20439?sk=24c4db0f925b9b5f4c0aa95100489807) By [Udaykiran Estari](https://medium.com/@UdaykiranEstari?source=post_page---byline--736b2fb20439-----------------------------------------)\n\nAgents can learn to satisfy a reward signal without completing the task the reward was meant to measure. This article connects a simple grid-world example to cases where a research model manipulated tests, monitoring, or reward infrastructure to make failures appear successful, then explains why standard audits can miss that behavior. It closes with practical defenses including inoculation prompts, reasoning-trace monitoring, hardened verification, and trajectory scanning.\n\n3. [mHC: Manifold-Constrained Hyper-Connections. Explained Through Equations, Architecture, Code and Visual Workflow](https://pub.towardsai.net/mhc-manifold-constrained-hyper-connections-explained-through-equations-architecture-code-eb516948d166?sharedUserId=tai-tech) By [JAIGANESAN](https://medium.com/@jaiganesan?source=post_page---byline--eb516948d166-----------------------------------------)\n\nByteDance’s Hyper-Connections widens the single residual stream into multiple learned copies, lifting benchmark scores but destabilizing a 27B mixture-of-experts run with a loss surge and gradient spike near step 12,000, driven by unconstrained mixing matrices amplifying signals up to 3,000x. mHC projects those matrices onto the doubly stochastic manifold using Sinkhorn-Knopp, capping amplification near 1.6 while preserving most of the accuracy gains. This article walks through the equations, architecture, and implementation, including the training instability that motivated the constraint and how mHC changes that behavior.\n\n4. [Automating Project Implementation and Maintenance with Claude Code + MCP + AI Agent Skills — Spectacular Productivity Gains, Surprising Drawbacks, and My Experience So Far](https://pub.towardsai.net/automating-project-implementaion-and-maintenance-with-claude-code-mcp-ai-agent-skills-213ac8e2a375?sharedUserId=tai-tech) By [Michalzarnecki](https://medium.com/@brightcode?source=post_page---byline--213ac8e2a375-----------------------------------------)\n\nGiving a coding agent access to logs, databases, issue trackers, and repositories lets it investigate problems across systems rather than from a single IDE context. This article shows how Claude Code, MCP servers, skills, subagents, and CLAUDE.md files can connect tools such as Jira, GitHub, Postgres, Elasticsearch, and Graylog, including a production incident where the agent correlated evidence across several sources. It also covers scheduled maintenance workflows and stresses that prompts and tool instructions should sit on top of real permission boundaries, not replace them.\n\nIf you are interested in publishing with Towards AI, [check our guidelines and sign up](https://contribute.towardsai.net/). We will publish your work to our network if it meets our editorial policies and standards.\n\n[LAI #144: Your Eval Improved. Did Your AI?](https://pub.towardsai.net/lai-144-your-eval-improved-did-your-ai-148ce2a3c7f7) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/lai-144-your-eval-improved-did-your-ai", "canonical_source": "https://pub.towardsai.net/lai-144-your-eval-improved-did-your-ai-148ce2a3c7f7?source=rss----98111c9905da---4", "published_at": "2026-09-24 16:16:01+00:00", "updated_at": "2026-09-24 16:31:05.170798+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "ai-products"], "entities": ["Towards AI", "Louis-François Bouchard", "Research Operating System", "Fadlih6481", "Adobe", "Adobe for Claude", "Claude", "ChatGPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lai-144-your-eval-improved-did-your-ai", "markdown": "https://wpnews.pro/news/lai-144-your-eval-improved-did-your-ai.md", "text": "https://wpnews.pro/news/lai-144-your-eval-improved-did-your-ai.txt", "jsonld": "https://wpnews.pro/news/lai-144-your-eval-improved-did-your-ai.jsonld"}}