{"slug": "you-changed-one-line-of-your-agent-s-prompt-what-else-broke", "title": "You changed one line of your agent's prompt. What else broke?", "summary": "A developer argues that AI agents need automated eval suites because non-deterministic behavior means a single prompt change can silently break unrelated features. The post recommends grading an agent's outcome, tool-call trajectory, and resulting state with plain code first, reserving LLM judges for subjective checks, and notes that a task passing 90% of the time only passes five consecutive trials about 59% of the time.", "body_md": "You tweak one line of your agent's prompt to fix a complaint. The fix works. You ship it.\n\nTwo days later, a different user reports something that used to work just fine.\n\nNobody touched that feature. You changed the prompt, and the agent's behavior shifted everywhere at once. That's why AI agents need evals, and why the evals most teams write miss the bugs that matter.\n\n**TL;DR**\n\nAn eval gives the agent a task, runs it, and scores the result automatically. Here's one for a scheduling agent:\n\n```\ntask: \"Book 30 minutes with Priya next Tuesday afternoon\"\nexpect:\n  - create_event called once\n  - attendee is priya@example.com\n  - start time is Tuesday, between 12:00 and 17:00\n  - no existing meeting was moved or deleted\ntrials: 5\npass: all 5 trials\n```\n\nRun your suite after every prompt, model, or tool change, and you'll know what broke before your users do.\n\nMost eval suites only check the final reply. But an agent can say \"Booked!\" while calling the wrong tool, or call the right tool with the wrong date. Check all three: the **outcome**, the **trajectory** (tool calls and their arguments), and the **state** that changed.\n\nStart with plain code. It's fast, free, and never disagrees with itself:\n\n``` python\ndef grade(trace, calendar):\n    events = [c for c in trace.tool_calls if c.name == \"create_event\"]\n    assert len(events) == 1\n    assert \"priya@example.com\" in events[0].args[\"attendees\"]\n    assert calendar.unchanged_except(events[0].result_id)\n```\n\nUse an LLM judge only for things code can't check, like tone or whether an answer is actually helpful. Then have a human review a sample of the judge's scores now and then, so you know the judge is grading the way you would.\n\nAgents are non-deterministic. If a task passes 90% of the time, the chance it passes **five times in a row** is only 59%. For anything user-facing, \"it worked when I tried it\" isn't good enough. Measure how often it works every time.\n\nDon't invent test cases. Mine them:\n\nTwenty real cases beat two hundred made-up ones.\n\n**Regression evals** check things that already work. They should pass almost 100% of the time, and a failure blocks the release.\n\n**Capability evals** check things the agent can't do reliably yet. A low score is expected. Watching it climb tells you whether your changes are actually helping.\n\n**An eval suite is the agent's memory of every mistake it has made.** Without one, every prompt change is a guess.\n\nHow do you decide when an agent change is safe to ship? Share in the comments.", "url": "https://wpnews.pro/news/you-changed-one-line-of-your-agent-s-prompt-what-else-broke", "canonical_source": "https://dev.to/vivek_1122/you-changed-one-line-of-your-agents-prompt-what-else-broke-196k", "published_at": "2026-10-15 07:43:15+00:00", "updated_at": "2026-10-15 07:46:19.225225+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/you-changed-one-line-of-your-agent-s-prompt-what-else-broke", "markdown": "https://wpnews.pro/news/you-changed-one-line-of-your-agent-s-prompt-what-else-broke.md", "text": "https://wpnews.pro/news/you-changed-one-line-of-your-agent-s-prompt-what-else-broke.txt", "jsonld": "https://wpnews.pro/news/you-changed-one-line-of-your-agent-s-prompt-what-else-broke.jsonld"}}