Wayfinder makes a decent case for how to actually structure AI

Wayfinder, an open-source reference implementation on GitHub, demonstrates a structured approach to AI evaluation, advocating for a pipeline that layers rule-based checks, LLM-as-a-judge, offline golden datasets, and online user signals rather than choosing a single method. The project highlights the risk of LLM-as-a-judge bias when the same model generates and evaluates, and provides code for component and end-to-end testing to help developers avoid regressions from prompt tweaks.

Wayfinder makes a decent case for how to actually structure AI The main problem is that most "guides" treat Rule-Based Eval, LLM-as-a-Judge, and End-to-End Eval as a menu where you just pick one. In reality, you need a pipeline. Wayfinder uses a single AI application as a consistent thread to demonstrate how to move from simple assertions to more complex offline and online evaluations. How to set up an evaluation strategy from scratch If you're trying to move past basic prompt tweaking, you can follow the logic implemented in Wayfinder to build a proper AI workflow. It's less about a specific tool and more about the architecture of your tests. 1. Start with Rule-Based Eval: This is your baseline. If the output must be JSON or must contain a specific keyword, use regex or schema validation. It's fast and deterministic. 2. Layer in LLM-as-a-Judge: For things like "tone" or "helpfulness" that regex can't catch, use a stronger model like GPT-4o or Claude /en/tags/claude/ 3.5 Sonnet to grade the output of your smaller production model. 3. Run Offline Evaluation: Create a golden dataset of 50-100 "ground truth" pairs. Every time you change a prompt, run the entire set to see if you've introduced regressions. 4. Move to Online Evaluation: This is the hardest part. You need to track real-world signals like thumbs up/down or conversion rates to see if the offline wins actually translate to user satisfaction. The technical breakdown of eval types Since this is a reference implementation, the value is in seeing these concepts in code. Here is how the different components usually break down in a real-world deployment: - Component Evaluation: Testing a single RAG /en/tags/rag/ step e.g., "Did the retriever actually find the right document?" - End-to-End Evaluation: Testing the final response given the initial query e.g., "Is the final answer correct regardless of the retrieval step?" - Offline vs Online: Offline is your "lab" testing; online is your "wild" testing. One thing that often gets overlooked is the "LLM-as-a-Judge" bias. If you use the same model to generate and evaluate, it tends to be overly optimistic about its own mistakes. The Wayfinder approach suggests using a diverse set of evaluation techniques to triangulate the truth. For anyone trying to implement this, I'd suggest starting with a small set of "unit tests" for your prompts. If you're using Python, you can easily script a loop that runs your prompt against a CSV of test cases and flags any output that fails a basic length or keyword check. This prevents the "one-step-forward, two-steps-back" cycle where fixing one edge case breaks ten other things. The whole project is available on GitHub if you want to see the specific code structure for these evals: https://github.com/wayfinder-ai/wayfinder It's a solid starting point for anyone who's tired of guessing whether their latest prompt tweak actually improved the app or just changed the flavor of the hallucinations. Next Can HydraFusion actually beat Claude Opus 5 on coding tasks? → /en/threads/8885/