Evals before vibes

Here's how a lot of prompt changes ship. You tweak the prompt, try it on three examples you remember, the answers look better, and you ship it. That's vibes.

I've done it too. Then I spent a few months building checks for my WhatsApp assistant, and the most useful thing they taught me was where my own checks were weaker than I thought. This post is what I built, what it caught, and a small demo that shows the difference between trying a prompt on a few examples and running it against a fixed set.

There's also a practical reason this is timely. OpenAI's hosted Evals platform became read only for existing users on 31 October 2026, and its dashboard and API are scheduled to shut down on 30 November. If your test cases and graders live there, this is a good time to move them somewhere you can version and run yourself. OpenAI's deprecation notes have the current dates.

Message Yourself OS reads my WhatsApp and tells me who's waiting on me and what I promised. Being wrong is expensive there: a bot that says "you promised this" had better be right. So most of the work went into checks.

Every finding carries its evidence. The model has to return the exact quote, the chat and the message it came from. Plain code then fetches that message and checks that the quote is really in it and the time matches. A finding ends up in one of a few states: verified, missing, not found, quote mismatch or time mismatch. Only verified findings reach the Focus list at the top of !today. Anything else appears under "Needs verification", never as fact.

58 test files pin the parsers, the evidence check and the rule that unverified items stay out of Focus.

A local evaluation runner has a fast tier that needs no WhatsApp connection or network, and a full tier that runs commands through a local server and tracks latency and regressions.

An AI judge scores replies. If the judgement itself fails, the result is reported as "not scored" and is never shown as a number.

A gate on real use. The newer, ranked !today stays behind a switch until it passes seven reviewed days of real use, with no errors and nothing missed.

!correct records why an extraction was wrong, in my own words.

The evidence check earned its keep straight away. It's the reason a made up quote can't turn into a reminder. It's also the part I trust most, because it's plain code checking plain facts.

The rest taught me more by showing me their limits:

!correct records reasons, but nothing reads them back. Pattern dikh raha hai? Everything that worked compared the output against something fixed: a real message, a test or a calendar of reviewed days. Everything that didn't was either checking itself or never run.

To show the idea without using my private chats, I built a small eval with 24 synthetic cases. Each case contains one to three chats and the expected findings written down by hand. The set has 17 expected findings in total, and nine cases should return nothing. It covers plain questions, requests without question marks, Hinglish questions and promises, group messages addressed to someone else, requests already answered, promises already kept, rhetorical questions and messages that should not count.

The task is the same as in my last post: find what's waiting on me and what I promised, with exact quotes. The code, cases, prompts and results are in the companion demo on GitHub.

I compared two prompts:

First, the vibes check. I ran both prompts on three easy cases, the kind you'd try by hand. Both got all three right in every run. If that's all you look at, v1 is fine.

Then I ran both prompts on all 24 cases using openai/gpt-oss-120b through Groq. I repeated the run three times. Each sequence in the table is in run order: one, two, then three.

A case only passes when the expected findings are present, nothing extra is returned, the finding has the right kind and its quote passes the evidence check. One bad finding can make the whole case fail.

The difference is in the cases you wouldn't think to try. v1 said a client's "Can you send the invoice?" was still waiting on me, two messages after I'd sent it. It flagged "Why is Bengaluru traffic like this every single day?" as a question for me. And it marked Arjun's "Bro the API docs link is broken" as waiting on me, the same mistake I found by eye in my last post. This time a test case catches it every run.

v2 isn't perfect either. The rhetorical question fooled it in every run, and in one run it missed a Hinglish promise that v1 got right. That's the point of a fixed set: you see exactly which cases moved, in which direction, instead of relying on a feeling that it's better.

I also ran an AI judge over the findings the prompts returned. It saw each finding together with the chat it came from, and had to give a pass or fail: was this really something waiting on Sayam, or really a promise he made? I then compared those judgements with the labels I had written by hand.

In every v1 run, the judge passed six findings that my labels said were wrong. It never failed one of the correct findings it judged. The examples included the broken link, the rhetorical question and the invoice I had already sent. Its explanations sounded reasonable: a broken link "requires Sayam to respond or fix"; a rhetorical question "legitimately expects a reply".

There is an important limit here: the judge only sees findings that the prompt returned. It cannot catch a missed promise, because there is no finding for it to judge. Its agreement is not a second score for the prompt and it is not a measure of recall.

This also reminded me of the limitation in my first production judge, which only saw the generated reply and not the messages behind it. The setups are different, but the lesson is similar: a plausible explanation is not evidence.

This is why I wouldn't trust an LLM judge just because it returns a confident explanation. Anthropic recommends calibrating judges against human experts, and the 2023 LLM as a judge study documented position, verbosity and self enhancement biases. A judge can be useful at scale, but its disagreements need to be checked against labels for your own task.

You don't need an evaluation platform to start. My demo is a JSONL file, two graders and a loop. The TypeScript runner has no package dependencies; a real model run still needs a provider API key and has to respect that provider's limits.

If you do want a tool, OpenAI's migration guide points to Promptfoo. In March 2026 Promptfoo announced it had agreed to be acquired by OpenAI and said it would remain open source. Inspect AI is another open source option, developed by the UK AI Security Institute and Meridian Labs. And this isn't just my concern. In LangChain's State of Agent Engineering survey, 52.4% of organisations surveyed reported running offline evaluations on test sets. It's a useful signal about the teams in that survey, not a measurement of every company building AI.

These are 24 synthetic cases, labelled by me, run three times on one model. They show how two prompts behaved on this particular set. They don't prove that v2 is better on real conversations, on another model or on a different task. My labels are judgement calls too: you could argue that a broken link really is waiting on me, and the judge did.

The judge results have a narrower meaning still. It only judges returned findings, so it cannot say anything about missed items. This is a small check of where the judge agrees with my labels, not a measure of recall or a production quality score.

The point isn't that v2 is the best prompt. It's that with a fixed set I can see which cases it fixed, which one it still gets wrong and which one it broke once. Then I can argue about specific cases instead of a feeling that a prompt is better.

Vibes are how you find what to test. Evals are how you know what changed.