{"slug": "i-have-been-vibecoding-evals-works-better-than-i-thought", "title": "I have been Vibecoding Evals (works better than I thought)", "summary": "A developer experimenting with AI coding agents found that adding evals to a support-triage app caught a subtle misclassification that manual testing missed. The app, built for a fictional shipment-tracking company, incorrectly escalated a routine API key rotation request as a security incident. Using the DeepEval agent skill, the developer created a golden dataset and ran automated checks, which pinpointed an over-broad prompt instruction. After fixing the prompt, the same test cases confirmed the fix without breaking other behaviors.", "body_md": "I’ve been building AI apps with coding agents for a while.\n\nLately, I’ve been experimenting with evals too.\n\nThe app in this example mostly worked. That was the problem.\n\nI built a small support-triage app for a fictional shipment-tracking company.\n\nA customer sends a support ticket, and the app decides what it is about, how urgent it is, and whether a human needs to respond.\n\nA real outage should be escalated.\n\nBut this ticket was different:\n\n“URGENT need key rotation now”\n\nThe customer was asking how to rotate their own API key before a security review.\n\nThe app classified it as a security incident and escalated it to a human.\n\nThat was wrong. The policy said normal key rotation was a self-service how-to request.\n\nNothing crashed. The app returned valid JSON. The fields all contained allowed values.\n\nThe behavior was still wrong.\n\nI could test a few tickets manually and convince myself the app worked.\n\nBut after changing the prompt, what would I actually know?\n\nWould the outage case still escalate?\n\nWould normal how-to questions stay in the normal queue?\n\nWould another API-key question behave differently?\n\nI didn’t want to change the prompt and simply hope for the best.\n\nI wanted a set of cases I could run again.\n\nI installed the DeepEval agent skill:\n\n`npx skills add confident-ai/deepeval --skill \"deepeval\"`\n\nThen I asked Cursor to add evals to the app:\n\n```\nThis app sometimes treats normal support questions like emergencies and sends\nthem to a human.\nAdd DeepEval so I can test this using the tickets and policy already in the repo.\nI am new to evals, so use the simplest setup DeepEval already provides, explain\nwhat you create, and ask me anything you need.\nRun the app as it is first and show me what fails. Do not fix it yet.\n```\n\nCursor already had the app, tickets, and policy, so it went straight to creating the baseline.\n\nThe first useful artifact was a JSON dataset.\n\nEach golden contained:\n\nthe customer message\n\nthe expected category\n\nthe expected priority\n\nwhether a human should be involved\n\nFor example:\n\n```\n{\n  \"input\": \"URGENT need key rotation now\",\n  \"expected\": {\n    \"category\": \"how-to\",\n    \"priority\": \"P3\",\n    \"needs_human\": false\n  }\n}\n```\n\nThe app does not learn from these examples.\n\nThe dataset is a checklist. It tells me which behaviors need to keep working before I ship a prompt update.\n\nThat was the part that made evals click for me.\n\nCursor used built-in DeepEval metrics to check two different things.\n\nThe first compared the app’s routing fields with the expected values in the golden:\n\nSo I could ask two separate questions:\n\nThe first run caught the key-rotation case.\n\nThe app returned:\n\n```\ncategory: security\npriority: P1\nneeds_human: true\n```\n\nThe expected result was:\n\n```\ncategory: how-to\npriority: P3\nneeds_human: false\n```\n\nThe output was valid. The behavior was not.\n\nThe policy check also gave Cursor a reason explaining why the decision did not fit the support rules.\n\nThat was already more useful than changing the prompt, clicking the same ticket again, and hoping I hadn’t broken something else.\n\nThe failure pointed back to an over-broad instruction in the triage prompt.\n\nIt was treating anything involving API keys as a security incident.\n\nThat was too aggressive.\n\nThe fix was to reserve escalation for actual compromise, suspicious access, or an outage.\n\nThen Cursor reran the same test cases.\n\nThe key-rotation case passed.\n\nThe real outage case still escalated.\n\nThat second part matters. Fixing one case is not enough. I also want to know that I didn’t break a case that was already working.\n\nThis is the workflow I’ve been using:\n\n```\nBuild the app\nCreate test cases\nRun the app\nRead the failure\nFix the prompt\nRun the same cases again\n```\n\nWithout evals, I would change the prompt and hope I didn’t create three new bugs.\n\nWith evals, every prompt change has a checklist behind it.\n\nI recorded the full workflow here:\n\nWatch the video: [https://x.com/juampitech/status/2084330588593385637?s=20](https://x.com/juampitech/status/2084330588593385637?s=20)\n\nDeepEval is open source and runs locally. You don’t need a Confident AI account for this workflow. LLM-as-a-judge metrics require a model API key.\n\nDeepEval Vibe Coder quickstart:", "url": "https://wpnews.pro/news/i-have-been-vibecoding-evals-works-better-than-i-thought", "canonical_source": "https://dev.to/juampitech/i-have-been-vibecoding-evals-works-better-than-i-thought-33oi", "published_at": "2026-08-03 20:38:30+00:00", "updated_at": "2026-08-03 21:43:15.320612+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "machine-learning"], "entities": ["DeepEval", "Cursor", "confident-ai"], "alternates": {"html": "https://wpnews.pro/news/i-have-been-vibecoding-evals-works-better-than-i-thought", "markdown": "https://wpnews.pro/news/i-have-been-vibecoding-evals-works-better-than-i-thought.md", "text": "https://wpnews.pro/news/i-have-been-vibecoding-evals-works-better-than-i-thought.txt", "jsonld": "https://wpnews.pro/news/i-have-been-vibecoding-evals-works-better-than-i-thought.jsonld"}}