The main problem is that most "guides" treat Rule-Based Eval, LLM-as-a-Judge, and End-to-End Eval as a menu where you just pick one. In reality, you need a pipeline. Wayfinder uses a single AI application as a consistent thread to demonstrate how to move from simple assertions to more complex offline and online evaluations.
How to set up an evaluation strategy from scratch #
If you're trying to move past basic prompt tweaking, you can follow the logic implemented in Wayfinder to build a proper AI workflow. It's less about a specific tool and more about the architecture of your tests.
-
Start with Rule-Based Eval: This is your baseline. If the output must be JSON or must contain a specific keyword, use regex or schema validation. It's fast and deterministic.
-
Layer in LLM-as-a-Judge: For things like "tone" or "helpfulness" that regex can't catch, use a stronger model (like GPT-4o or Claude 3.5 Sonnet) to grade the output of your smaller production model.
-
Run Offline Evaluation: Create a golden dataset of 50-100 "ground truth" pairs. Every time you change a prompt, run the entire set to see if you've introduced regressions.
-
Move to Online Evaluation: This is the hardest part. You need to track real-world signals (like thumbs up/down or conversion rates) to see if the offline wins actually translate to user satisfaction.
The technical breakdown of eval types #
Since this is a reference implementation, the value is in seeing these concepts in code. Here is how the different components usually break down in a real-world deployment:
- Component Evaluation: Testing a singleRAG step (e.g., "Did the retriever actually find the right document?")
- End-to-End Evaluation: Testing the final response given the initial query (e.g., "Is the final answer correct regardless of the retrieval step?")
- Offline vs Online: Offline is your "lab" testing; online is your "wild" testing.
One thing that often gets overlooked is the "LLM-as-a-Judge" bias. If you use the same model to generate and evaluate, it tends to be overly optimistic about its own mistakes. The Wayfinder approach suggests using a diverse set of evaluation techniques to triangulate the truth.
For anyone trying to implement this, I'd suggest starting with a small set of "unit tests" for your prompts. If you're using Python, you can easily script a loop that runs your prompt against a CSV of test cases and flags any output that fails a basic length or keyword check. This prevents the "one-step-forward, two-steps-back" cycle where fixing one edge case breaks ten other things.
The whole project is available on GitHub if you want to see the specific code structure for these evals:
https://github.com/wayfinder-ai/wayfinder
It's a solid starting point for anyone who's tired of guessing whether their latest prompt tweak actually improved the app or just changed the flavor of the hallucinations.
Next Can HydraFusion actually beat Claude Opus 5 on coding tasks? →