# Wayfinder makes a decent case for how to actually structure AI > Source: > Published: 2026-09-06 17:46:12+00:00 # Wayfinder makes a decent case for how to actually structure AI The main problem is that most "guides" treat Rule-Based Eval, LLM-as-a-Judge, and End-to-End Eval as a menu where you just pick one. In reality, you need a pipeline. Wayfinder uses a single AI application as a consistent thread to demonstrate how to move from simple assertions to more complex offline and online evaluations. ## How to set up an evaluation strategy from scratch If you're trying to move past basic prompt tweaking, you can follow the logic implemented in Wayfinder to build a proper AI workflow. It's less about a specific tool and more about the architecture of your tests. 1. **Start with Rule-Based Eval:** This is your baseline. If the output must be JSON or must contain a specific keyword, use regex or schema validation. It's fast and deterministic. 2. **Layer in LLM-as-a-Judge:** For things like "tone" or "helpfulness" that regex can't catch, use a stronger model (like GPT-4o or [Claude](/en/tags/claude/) 3.5 Sonnet) to grade the output of your smaller production model. 3. **Run Offline Evaluation:** Create a golden dataset of 50-100 "ground truth" pairs. Every time you change a prompt, run the entire set to see if you've introduced regressions. 4. **Move to Online Evaluation:** This is the hardest part. You need to track real-world signals (like thumbs up/down or conversion rates) to see if the offline wins actually translate to user satisfaction. ## The technical breakdown of eval types Since this is a reference implementation, the value is in seeing these concepts in code. Here is how the different components usually break down in a real-world deployment: - **Component Evaluation:** Testing a single[RAG](/en/tags/rag/) step (e.g., "Did the retriever actually find the right document?") - **End-to-End Evaluation:** Testing the final response given the initial query (e.g., "Is the final answer correct regardless of the retrieval step?") - **Offline vs Online:** Offline is your "lab" testing; online is your "wild" testing. One thing that often gets overlooked is the "LLM-as-a-Judge" bias. If you use the same model to generate and evaluate, it tends to be overly optimistic about its own mistakes. The Wayfinder approach suggests using a diverse set of evaluation techniques to triangulate the truth. For anyone trying to implement this, I'd suggest starting with a small set of "unit tests" for your prompts. If you're using Python, you can easily script a loop that runs your prompt against a CSV of test cases and flags any output that fails a basic length or keyword check. This prevents the "one-step-forward, two-steps-back" cycle where fixing one edge case breaks ten other things. The whole project is available on GitHub if you want to see the specific code structure for these evals: ``` https://github.com/wayfinder-ai/wayfinder ``` It's a solid starting point for anyone who's tired of guessing whether their latest prompt tweak actually improved the app or just changed the flavor of the hallucinations. [Next Can HydraFusion actually beat Claude Opus 5 on coding tasks? →](/en/threads/8885/)