AI Evals: Everything You Need to Know Hamel Husain and Shreya Shankar published an AI Evals FAQ distilling the most common questions from teaching 700+ engineers and product managers about AI evaluation. The guide distinguishes model benchmarks such as GPQA Diamond, Terminal-Bench, and MMLU from product evals, which measure whether a specific AI product — including its model, prompts, retrieval, tools, and application code — does what users and the business need. It recommends analyzing traces to find real failure modes, converting important failures into targeted evals, and rerunning those evals to guide changes. This document curates the most common questions Shreya and I received while teaching https://maven.com/parlance-labs/evals?promoCode=evals-info-book 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. Browse the questions that interest you, or choose a guide below for a curated reading path through the FAQs and related articles. | Where are you? | This sounds like me | |---|---| | I’m new to evals ../../../notes/llm/evals/start/new-to-evals/index.html | I’ve heard the term, but I’m not sure what evals involve or whether I need them. | | I don’t know what to test ../../../notes/llm/evals/start/getting-started/index.html | I’m building an AI product, but I haven’t figured out which failures to measure or what good performance looks like. | | I don’t trust my eval scores ../../../notes/llm/evals/start/trust-your-evals/index.html | We have evals, but the scores don’t match our judgment of the outputs, or tests pass while users still encounter problems. | | My product feels too hard to evaluate ../../../notes/llm/evals/start/hard-to-evaluate/index.html | Our outputs are subjective, long, or involve many steps. Even a knowledgeable person has trouble deciding whether they’re right. | | Evals take too much time or money ../../../notes/llm/evals/start/reduce-eval-cost/index.html | We’re spending too much effort reviewing outputs, maintaining tests, or running evaluators. | Browse all questions by section. AI evals are tests that tell you whether an AI system is doing what you want. They give your team feedback when the product drifts from user needs or business goals. The failures they catch also become data you can use to improve the system. More formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or structured review. Most AI products need several evals because they can fail in different ways. When you hear the word “evals,” it usually refers to one of two things: model benchmarks or product evals. Model benchmarks compare general-purpose models on shared tasks. Model providers publish these benchmark results when they release new models. Common examples include GPQA Diamond for graduate-level science reasoning, Terminal-Bench for agents doing complex work in command-line environments, and MMLU for knowledge and reasoning across a wide range of subjects. These scores can help you choose a promising model as a starting point. To assess quality on your own tasks you need product evals, which we discuss next. Product evals measure whether your specific AI product does what you want it to do. They turn your judgment about what a good product experience looks like into metrics you can track. Product evals encompass all components of your product, including the model, prompts, retrieval, tools, and application code. This flavor of evals are focused on capturing failures that matter to users and the business. Consider an order-cancellation agent. Its product evals might check whether it selected the correct order and waited for the cancellation tool to succeed before telling the user the order was canceled. A high score on GPQA Diamond or Terminal-Bench gives you little information on this, because those benchmarks don’t have access to your systems. There are several mechanisms you can use to implement product evals, including code assertions, human review, LLM judges, and online experiments. The right method depends on the failure being measured, and is discussed in greater detail in this series ../../../notes/llm/evals/index.html . In the rest of the AI Evals FAQ ../../../blog/posts/evals-faq/index.html , we focus on product evals. It starts with analyzing traces to discover real failure modes. We then turn important failures into targeted evals and use the results to guide changes. Finally, rerunning the evals ../../../blog/posts/evals/index.html step-3-run-track-your-tests-regularly tells us whether the system improved. If you are completely new to product-specific evals, see these posts: | | Guide | What it covers | |---|---|---| | | Part 1 ../../../blog/posts/evals/index.html : Your AI Product Needs Evals | Build a domain-specific evaluation system with scoped tests, trace review, human evaluation, and experiments. | | | Part 2 ../../../blog/posts/llm-judge/index.html : Using LLM-as-a-Judge For Evaluation: A Complete Guide | Capture a domain expert’s judgment, automate it with an LLM judge, and validate the judge against human labels. | | | Part 3 ../../../blog/posts/field-guide/index.html : A Field Guide to Rapidly Improving AI Products | Use error analysis, realistic data, and trustworthy evals to run a sustained product-improvement loop. | A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every step across all agents, tools, and system components in a session: multiple user messages, assistant responses, retrieved documents, and intermediate tool interactions. Note on terminology: Different observability vendors use varying definitions of traces and spans. Alex Strick van Linschoten’s analysis https://mlops.systems/posts/2025-06-04-instrumenting-an-agentic-app-with-arize-phoenix-and-litellm.html llm-tracing-tools-naming-conventions-june-2025 highlights these differences screenshot below : Start with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one domain expert who understands your users as your quality decision maker a “benevolent dictator” . Use a notebook to review traces and analyze data, or build your own custom annotation interface with an AI coding assistant like Claude or Codex. Either way, you can write arbitrary code, visualize data, and iterate quickly. The video https://youtu.be/aqKUwPKBkB0?si=5KDmMQnRzO Ce9xH below shows a simple annotation interface built inside a notebook. It’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development. You should always be doing error analysis https://www.youtube.com/watch?v=qH1dZ8JLLdU . When you discover issues through error analysis, many will be straightforward bugs you’ll fix immediately. These fixes don’t require separate evaluation infrastructure as they’re just part of development. The decision to build automated evaluators comes down to cost-benefit analysis. If you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it. But if you need to align an LLM-as-judge evaluator, consider whether the failure mode warrants that investment. In the projects we’ve worked on, we’ve spent 60-80% of our development time on error analysis and evaluation . Expect most of your effort to go toward understanding failures i.e. looking at data rather than building automated checks. Be wary of optimizing for high eval pass rates https://ai-execs.com/2 intro.html a-case-study-in-misleading-ai-advice . If you’re passing 100% of your evals, you’re likely not challenging your system enough. A 70% pass rate might indicate a more meaningful evaluation that’s actually stress-testing your application. Focus on evals that help you catch real issues, not ones that make your metrics look good. Yes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important. Today’s prompt engineering tricks might become obsolete, but you’ll still need to understand failure modes. Additionally, a LLM cannot read your mind, and research shows https://arxiv.org/abs/2404.12272 that people need to observe the LLM’s behavior in order to properly externalize their requirements. For deeper perspective on this debate, see these two viewpoints: “The model is the product” https://m.youtube.com/watch?si=qknrtQeITqJ7VsJH&v=4dUFIRj-BWo&feature=youtu.be versus “The model is NOT the product” https://www.youtube.com/watch?v=EEw2PpL- NM . Don’t try to sell your team on “evals”. Instead, show them what you find when you look at the data. Start by doing the error analysis yourself. Look at 50 to 100 real user conversations and find the most common ways the product is failing. Use these findings to tell a story with data. Present your team with: Frame evaluation as part of development, not optional testing. Keep a running log of the errors you catch, what you learned, the fix, and the likely impact you avoided. Share it weekly or monthly. A concrete report such as “we caught 47 issues before users saw them” makes the value easier to see than an abstract pitch about evals. This approach builds trust. Don’t just show dashboards and metrics; tell the story of what you’re finding in the data. By narrating your findings, you teach the team what you’re learning, providing immediate value. When you fix an issue, show how the error rate for that specific problem went down. Soon, your team will see the progress and ask how you’re doing it. Let results instead of methods lead the conversation. This is similar to classic machine learning projects, where outcomes are speculative and progress is bounded by iterating on experiments https://hamel.dev/blog/posts/field-guide/ your-ai-roadmap-should-count-experiments-not-features . In this situation, it’s important that you share the learnings from each experiment to show progress and encourage investment. Error analysis is the most important activity in evals . Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data. The process involves: Gathering representative traces of user interactions with the LLM. If you do not have any data, you can generate synthetic data to get started. Human annotator s ideally a benevolent dictator review and write open-ended notes about traces, noting any issues. This process is akin to “journaling” and is adapted from qualitative research methodologies. Start by annotating at least 30 traces yourself before reviewing suggestions from an agent. When beginning, it is recommended to focus on noting the first failure observed in a trace, as upstream errors can cause downstream issues, though you can also tag all independent failures if feasible. A domain expert https://hamel.dev/blog/posts/llm-judge/ step-1-find-the-principal-domain-expert should be performing this step. Categorize the open-ended notes into a “failure taxonomy.” In other words, group similar failures into distinct categories. Axial coding is the most important step. At the end, count the number of failures in each category. You can use an LLM to help with this step. Have your agent cluster the data and choose a diverse initial sample. After your first 30 annotations, let it search the remaining traces for likely instances of the failures you described. Accept or reject its suggestions and keep iterating until you reach theoretical saturation https://delvetool.com/blog/theoreticalsaturation , meaning new reviews stop revealing failure modes or changing existing ones. A working pool of roughly 100 diverse traces is a useful guardrail for this human-agent loop. The agent can focus your attention on the most informative traces, so you no longer have to read all 100 sequentially. See how many examples you need for each kind of eval for the full breakdown. You should frequently revisit this process. There are advanced ways to sample data more efficiently how-can-i-efficiently-sample-production-traces-for-review.html , like clustering, sorting by user feedback, and sorting by high probability failure patterns. Over time, you’ll develop a “nose” for where to look for failures in your data. Do not skip error analysis. It ensures that the evaluation metrics you develop are supported by real application behaviors instead of counter-productive generic metrics which most platforms nudge you to use . For examples of how error analysis can be helpful, see this video https://www.youtube.com/watch?v=e2i6JbU2R-s , or this blog post https://hamel.dev/blog/posts/field-guide/ . Here is a visualization of the error analysis process by one of our students, Pawel Huryn https://www.linkedin.com/in/pawel-huryn/ - including how it fits into the overall evaluation process: No. Writing a rubric before you review examples can get in the way. Let’s get some definitions out of the way: Both can help, but treat your initial expectations as a starting point that you will revise. It’s often better to wait until you’ve reviewed some examples before developing a detailed rubric. Reviewers can become so focused on checking each item that they overlook problems outside the rubric. It’s important to give reviewers room to notice things you didn’t anticipate. This change in what you consider good is called “criteria drift” https://arxiv.org/abs/2404.12272 . For example, let’s say you have a support agent that handles refunds and it escalates refunds to a human per your policy. You might only realize that the process is frustrating for the user after reading a few interactions. Don’t underestimate the degree of criteria drift that will happen as you review examples We recommend using error analysis ../../../blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html to systematically review examples and decide what might belong in the rubric. This involves writing open-ended notes about what looks wrong, then group similar notes to see which problems recur. See this live demo https://www.youtube.com/watch?v=BsWxPI9UM4c for a walkthrough. After doing error analysis, you can write a better rubric informed by user and application behavior. You should periodically ../../../blog/posts/evals-faq/how-often-should-i-re-run-error-analysis-on-my-production-system.html do error analysis to make sure your rubric is current. Yes. When reviewing interactions, write down anything that makes the product less useful. This includes missing or broken features that have nothing to do with the model. Additionally, don’t focus on why the error occurred, as that should only come after you prioritize which issues to fix. For example, a support agent might tell a customer that an order has shipped without providing a tracking link. Even if your AI doesn’t have the ability to fetch a tracking link, record that problem. A prerequisite to building evals is to identify and prioritize which issues to fix through error analysis ../../../blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html . Some of these issues may end up being engineering or design issues that don’t need ../../../blog/posts/evals-faq/should-i-build-automated-evaluators-for-every-failure-mode-i-find.html an automated evaluator, but they are still important to fix Lastly, we’ve found that deferring root-cause analysis and focusing on problems allows you to write higher-quality annotations while looking at more data. Building evals is a pipeline, and each stage needs a different amount of data. We describe these stages below: | Stage | What to do | |---|---| | 1. Review the application | Read traces and write down the ways your application fails. This process is called error discovery ../../../blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html . Start with 100 diverse traces and annotate at least the first 30 yourself. | | 2. Create and validate evaluators | Choose between two evaluator types. Use a code-based eval ../../../blog/posts/evals/index.html level-1-unit-tests when an objective rule can identify the failure. Include Pass and Fail examples for every condition and important edge case. Use an LLM judge ../../../blog/posts/llm-judge/index.html when the failure requires human judgment. Label 100 to 200 examples for each failure mode. | | 3. Build a repeatable eval set | Collect examples that represent important workflows and confirmed failures. Run this set when you change your application. These sets often grow to 100 or more examples. | A trace is a complete record of one user session with your application. Ask a coding agent to help you sample the initial pool so it covers different users and workflows. Our evals plugin https://github.com/ai-evals-course/evals-skills can help with sampling and build an annotation interface for your traces. We recommend annotating at least 30 traces with a process called error discovery ../../../blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html yourself before asking the agent to suggest failures. Write free-text notes about anything that seems wrong from the user’s perspective. These examples give the agent a concrete record of your judgment. Keep this first pass manual. If the agent starts suggesting problems too early, its guesses can bias your judgment. You may miss failures that depend on product context or your definition of a good user experience. After 30 traces, ask the agent to search the remaining pool for similar examples. Review every suggestion yourself. Accept or reject each one and correct the agent when it misunderstands your criteria. Continue until new traces stop revealing failure modes or changing existing ones. Qualitative researchers call this theoretical saturation https://delvetool.com/blog/theoreticalsaturation . We recommend reviewing at least 100 traces. Continue past 100 while you are still learning. If you want to see the process done live, watch the live walkthrough https://youtu.be/tqUDjc1HzO4 . The video shows how Shreya Shankar uses an agent to review traces quickly while keeping a human in charge of the failure criteria. This review produces a failure taxonomy, which is a list of the specific ways your application fails. Use that taxonomy to decide which evaluators to build. Choose an evaluator for each important failure mode. The evaluator type determines how many labeled examples you need. Code-based evals work for objective rules. LLM judges work for failures that require human judgment. Use a code-based eval when a deterministic rule can identify the failure. Examples include checking whether JSON parses or whether a tool call uses the correct arguments. The number of examples depends on the scenarios the check covers. At minimum, include examples that should Pass and Fail for every condition. Add important edge cases you found during error discovery. A check with one rule may need only a few examples that Pass and a few that Fail. Use an LLM judge when the failure requires subjective or domain-specific judgment. Plan to label 100 to 200 examples for each failure mode. Reuse labeled traces from error discovery when they match the failure mode, then collect more until you reach that range. The labels should come from a trusted domain expert ../../../blog/posts/evals-faq/how-many-people-should-annotate-my-llm-outputs.html and contain enough Pass and Fail examples to evaluate both classes. Split these examples into train, dev, and test sets. Use 10 to 20 percent for train examples that may appear in the prompt. Use 40 to 45 percent for dev while refining the judge. Reserve the remaining 40 to 45 percent for one final test. When possible, include 30 to 50 Pass examples and 30 to 50 Fail examples in both the dev and test sets. The validation guide ../../../blog/posts/llm-judge/index.html how-do-you-validate-an-llm-judge-against-human-labels explains the full process. The judge-validation flashcard ../../../notes/llm/evals/flashcards/7-how-to-trust-a-llm-judge.png is a good visual reference as well. After validating the evaluators, assemble the examples you will run repeatedly during development. Start with examples from error discovery that capture important failure modes. Add confirmed failures as you find them. A purpose-built eval set often grows to 100 or more examples. Coverage determines the final size. Each important workflow and known failure should be represented, and the set should remain cheap enough to run often. Code-based checks and LLM judges can run over the same examples. The CI evals ../../../blog/posts/evals-faq/how-are-evaluations-used-differently-in-cicd-vs-monitoring-production.html FAQ explains how to use this set during development. While user feedback is a good way to narrow in on problematic traces, other methods are also useful. Here are three complementary approaches: The simplest approach is reviewing a random sample of traces. If you find few issues, escalate to stress testing: create queries that deliberately test your prompt constraints to see if the AI follows your rules. Use existing evals to find problematic traces and potential issues. Once you’ve identified these, you can proceed with the typical evaluation process starting with error analysis. For more sophisticated trace discovery, use outlier detection, metric-based sorting, and stratified sampling to find interesting traces. Generic metrics can serve as exploration signals to identify traces worth reviewing, even if they don’t directly measure quality. Re-run error analysis when making significant changes: new features, prompt updates, model switches, or major bug fixes. A useful heuristic is to set a goal for reviewing at least 100+ fresh traces each review cycle. Typical review cycles we’ve seen range from 2-4 weeks. See this FAQ on how to sample traces effectively. Between major analyses, review 10-20 traces weekly, focusing on outliers: unusually long conversations, sessions with multiple retries, or traces flagged by automated monitoring. Adjust frequency based on system stability and usage growth. New systems need weekly analysis until failure patterns stabilize. Mature systems might need only monthly analysis unless usage patterns change. Always analyze after incidents, user complaint spikes, or metric drift. Scaling usage introduces new edge cases. Eval datasets naturally get stale as your product and users change. Use regular error analysis ../../../blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html to find new problems and update your examples or reference answers. How often you review ../../../blog/posts/evals-faq/how-often-should-i-re-run-error-analysis-on-my-production-system.html depends on your use case and how quickly your product or usage changes. Like unit tests, evals can catch problems that return after a fix. However, evals often cost considerably more than unit tests to maintain and run. Therefore, you should weigh each eval’s cost against the value of its signals. If everything keeps passing ../../../blog/posts/evals-faq/how-much-of-my-development-budget-should-i-allocate-to-evals.html , this is a sign that the eval is no longer useful and should be retired or run less often. As your eval set changes, its scores may no longer be directly comparable with older scores. That is ok One purpose of evals are to provide you with challenges you can hill climb against. These challenges should change as your product evolves to help you keep improving. For tracking progress with metrics over longer time horizons, it’s often better to use product metrics in addition to evals. Examples of product metrics include: churn, active users, revenue, etc. A common mistake is prompting an LLM to "give me test queries" without structure, resulting in generic, repetitive outputs. A structured approach using dimensions produces far better synthetic data for testing LLM applications. Use synthetic data to start error analysis before you have enough production traffic, or to test a known failure that appears rarely in real data. Define the variation you need, generate examples, run them through the full system, and review the resulting traces. Synthetic data cannot tell you how common a failure is in production. It can also miss details that matter in specialized domains. Compare synthetic examples with real data as soon as real data becomes available. See when synthetic data may be unreliable ../../../blog/posts/evals-faq/are-there-scenarios-where-synthetic-data-may-not-be-reliable.html for cases that require extra review. Start by defining dimensions : categories that describe different aspects of user queries. Each dimension captures one type of variation in user behavior. For example: Start with failure hypotheses . If you lack intuition about failure modes, use your application extensively or recruit friends to use it. Then choose dimensions targeting those likely failures. Create tuples manually first : Write 20 tuples by hand. Each tuple selects one value from each dimension. Example: Vegan , Italian , Multi-step . This manual work helps you understand your problem space. Scale with two-step generation : This separation avoids repetitive phrasing. The Vegan , Italian , Multi-step tuple becomes: "I need a dairy-free lasagna recipe that I can prep the day before." You can generate tuples two ways: Cross product then filter : Generate all dimension combinations, then filter with an LLM. Guarantees coverage including edge cases. Use when most combinations are valid. Direct LLM generation : Ask the LLM to generate tuples directly. This produces more realistic combinations, but it tends toward generic outputs and misses rare scenarios. Use it when many dimension combinations are invalid. Fix obvious problems first : Don’t generate synthetic data for issues you can fix immediately. If your prompt doesn’t mention dietary restrictions, fix the prompt rather than generating specialized test queries. After iterating on your tuples and prompts, run these synthetic queries through your actual system to capture full traces . A pool of roughly 100 diverse traces is a useful starting point for failure discovery. Have an agent help with sampling, annotate at least 30 traces yourself, then review the agent’s suggestions until your learning plateaus. See how many examples you need for error discovery for the full explanation. Here is a visual ../../../notes/llm/evals/flashcards/9-synthetic-data.png that helps visualize the process. Yes: synthetic data can mislead or mask issues. For guidance on generating synthetic data when appropriate, see What is the best approach for generating synthetic data? Common scenarios where synthetic data fails: Complex domain-specific content : LLMs often miss the structure, nuance, or quirks of specialized documents e.g., legal filings, medical records, technical forms . Without real examples, critical edge cases are missed. Low-resource languages or dialects : For low-resource languages or dialects, LLM-generated samples are often unrealistic. Evaluations based on them won’t reflect actual performance. When validation is impossible : If you can’t verify synthetic sample realism due to domain complexity or lack of ground truth , real data is important for accurate evaluation. High-stakes domains : In high-stakes domains medicine, law, emergency response , synthetic data often lacks subtlety and edge cases. Errors here have serious consequences, and manual validation is difficult. Underrepresented user groups : For underrepresented user groups, LLMs may misrepresent context, values, or challenges. Synthetic data can reinforce biases in the training data of the LLM. There is no replacement for looking at real interactions. This situation is not ideal, but there are some things you can do. Here are some options, in order of preference: Try to find real data you are allowed to inspect. A customer may agree to share a subset of traces, or test users may let you review their interactions. Even limited access gives you examples of how people use the product. If you cannot inspect the data yourself, work with domain experts who are allowed to see it. Make your product easier for them to verify ../../../blog/posts/eval-smell/index.html as part of their normal work. For example, a medical research assistant could show a clinician the evidence behind each claim and flag conflicting sources for review. The clinician can correct a specific claim or resolve a conflict while using the product. Those decisions can provide additional data for evals, subject to the same restrictions on what you can store and share. To design this well, learn how the experts check an answer. Give them links to the supporting evidence and smaller pieces of work they can review. Asking whether the final answer was helpful often tells you too little about what went wrong. I discuss this approach in this post ../../../blog/posts/eval-smell/ . Redact or edit traces so they can be shared. If sensitive information cannot be stored, redact it before logging. Redaction tools can miss sensitive information, so check their output. When edited traces can be shared, removing personal information and changing sensitive details can make real examples usable for review. Check that those edits preserve the behavior you need to evaluate. If none of the above options are possible, synthetic data should be your last resort. Synthetic data can help you find initial problems but has the downside that it only gives you limited evidence about how real users will behave. Read more about when synthetic data may be unreliable ../../../blog/posts/evals-faq/are-there-scenarios-where-synthetic-data-may-not-be-reliable.html . Complex applications often support vastly different query patterns—from “What’s the return policy?” to “Compare pricing trends across regions for products matching these criteria.” Each query type exercises different system capabilities, leading to confusion on how to design eval criteria. Error Analysis https://youtu.be/e2i6JbU2R-s?si=8p5XVxbBiioz69Xc is all you need. Your evaluation strategy should emerge from observed failure patterns e.g. error analysis , not predetermined query classifications. Rather than creating a massive evaluation matrix covering every query type you can imagine, let your system’s actual behavior guide where you invest evaluation effort. During error analysis, you’ll likely discover that certain query categories share failure patterns. For instance, all queries requiring temporal reasoning might struggle regardless of whether they’re simple lookups or complex aggregations. Similarly, queries that need to combine information from multiple sources might fail in consistent ways. These patterns discovered through error analysis should drive your evaluation priorities. It could be that query category is a fine way to group failures, but you don’t know that until you’ve analyzed your data. To see an example of basic error analysis in action, see this video https://youtu.be/e2i6JbU2R-s?si=8p5XVxbBiioz69Xc . There are many ways to sample production traces for review. Here are some common methods. | Method | What it does | Main limitation | |---|---|---| | Random | Selects traces with equal probability. | A small batch can miss rare cases. | | Clustering | Groups traces by similar content and selects examples from each group. | The result depends on the features and clustering choices. | | Data analysis | Reviews extreme values such as latency or tool count. | An extreme value may have nothing to do with quality. | | Classification | Uses an evaluator or another model to flag likely failures. | It favors problems the classifier already knows how to find. | | Feedback | Selects traces with negative user feedback. | It misses problems that users do not report. | The table above orders sampling methods from the most exploratory to the most targeted. When you’re starting out, you should optimize for exploration of the data. As you learn more, you can start to lean more heavily on signals to select traces. The proper mix of methods depends on your goals and requires experimentation. Keep some random traces in every batch. This gives you a chance to find failure modes that your current signals do not describe. Use targeted sampling to find rare failures. Search for signals that correlate with the failure, such as a specific tool sequence, unusually long traces, retries, or a known input pattern. Review the targeted batch to collect examples and improve the failure definition. This flashcard ../../../notes/llm/evals/flashcards/8-sample-traces.png from our evals flashcards series visualizes these methods. We can borrow a technique from machine learning called active learning to sample production traces. In active learning, a system asks a person to label the data points that would be most useful for its next update. In Shreya Shankar’s walkthrough https://youtu.be/tqUDjc1HzO4 , Claude Code clusters traces and chooses examples from each cluster for review. A monitor command watches annotations.json for new labels. When a label arrives, the agent updates a failure taxonomy and looks for similar cases or different failures. In the above video, active learning is used in the context of error analysis to find new cases to review. However, this approach can be used anywhere in the workflow where you are annotating data. 👉 Want to learn more about AI Evals? Check out our AI Evals course https://maven.com/parlance-labs/evals?promoCode=evals-info-book . It’s a live cohort with hands on exercises and office hours. Here is a 25% discount code https://maven.com/parlance-labs/evals?promoCode=evals-info-book for readers. 👈 Engineers often believe that Likert scales 1-5 ratings provide more information than binary evaluations, allowing them to track gradual improvements. However, this added complexity often creates more problems than it solves in practice. Binary evaluations force clearer thinking and more consistent labeling. Likert scales introduce significant challenges: the difference between adjacent points like 3 vs 4 is subjective and inconsistent across annotators, detecting statistical differences requires larger sample sizes, and annotators often default to middle values to avoid making hard decisions. Having binary options forces people to make a decision rather than hiding uncertainty in middle values. Binary decisions are also faster to make during error analysis - you don’t waste time debating whether something is a 3 or 4. For tracking gradual improvements, consider measuring specific sub-components with their own binary checks rather than using a scale. For example, instead of rating factual accuracy 1-5, you could track “4 out of 5 expected facts included” as separate binary checks. This preserves the ability to measure progress while maintaining clear, objective criteria. Start with binary labels to understand what ‘bad’ looks like. Numeric labels are advanced and usually not necessary. Generally no. Eval-driven development writing evaluators before implementing features sounds appealing but creates more problems than it solves. Unlike traditional software where failure modes are predictable, LLMs have infinite surface area for potential failures. You can’t anticipate what will break. A better approach is to start with error analysis. Write evaluators for errors you discover, not errors you imagine. This avoids getting blocked on what to evaluate and prevents wasted effort on metrics that have no impact on actual system quality. Exception: Eval-driven development may work for specific constraints where you know exactly what success looks like. If adding “never mention competitors,” writing that evaluator early may be acceptable. Most importantly, always do a cost-benefit analysis before implementing an eval. Ask whether the failure mode justifies the investment. Error analysis reveals which failures actually matter for your users. Focus automated evaluators on failures that persist after fixing your prompts. Many teams discover their LLM doesn’t meet preferences they never actually specified - like wanting short responses, specific formatting, or step-by-step reasoning. Fix these obvious gaps first before building complex evaluation infrastructure. Consider the cost hierarchy of different evaluator types. Simple assertions and reference-based checks comparing against known correct answers are cheap to build and maintain. LLM-as-Judge evaluators require 100+ labeled examples, ongoing weekly maintenance, and coordination between developers, PMs, and domain experts. This cost difference should shape your evaluation strategy. Only build expensive evaluators for problems you’ll iterate on repeatedly. Since LLM-as-Judge comes with significant overhead, save it for persistent generalization failures - not issues you can fix trivially. Start with cheap code-based checks where possible: regex patterns, structural validation, or execution tests. Reserve complex evaluation for subjective qualities that can’t be captured by simple rules. No. Generic evaluations waste time and create false confidence when you use them as quality measures. However, they can still help you find traces to inspect. Generic evaluation metrics are everywhere. Eval libraries contain scores like helpfulness, coherence, quality, etc. promising easy evaluation. These metrics measure abstract qualities that may not matter for your use case. Good scores on them don’t mean your system works. Instead, conduct error analysis to understand failures. Define binary failure modes based on real problems. Create custom evaluators for those failures and validate them against human judgment. Experienced practitioners may use generic metrics as exploration signals. Once you understand why they fail as quality measures, you can use them to find interesting traces for human review. Generic metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using error analysis to identify metrics specific to your application’s behavior. We recommend designing binary pass/fail. evals using LLM-as-judge or code-based assertions. As an example, consider a real estate CRM assistant. Suggesting showings that aren’t available can be tested with an assertion or confusing client personas can be tested with a LLM-as-judge is problematic . Generic metrics like similarity or verbosity won’t catch this. A relevant quote from the course: “The abuse of generic metrics is endemic. Many eval vendors promote off the shelf metrics, which ensnare engineers into superfluous tasks.” Similarity metrics aren’t always useless. They have utility in domains like search and recommendation and therefore can be useful for optimizing and debugging retrieval for RAG . For example, cosine similarity between embeddings can measure semantic closeness in retrieval systems, and average pairwise similarity can assess output diversity where lower similarity indicates higher diversity . For LLM-as-Judge selection, using the same model is usually fine because the judge is doing a different task than your main LLM pipeline. While research has shown https://arxiv.org/pdf/2508.06709 that models can exhibit bias when evaluating their own outputs, what ultimately matters is how well your judge aligns with human judgments. The judges we recommend building do scoped binary classification tasks. We’ve found that iterative alignment with human labels is usually achievable on this constrained task. Focus on achieving high True Positive Rate TPR and True Negative Rate TNR with your judge on a held out labeled test set. If you struggle to achieve good alignment with human scores, then consider trying a different model. However onboarding new model providers may involve non-trivial effort in some organizations, which is why we don’t advocate for using different models by default unless there’s a specific alignment issue. When selecting judge models, start with the most capable models available to establish strong alignment with human judgments. You can optimize for cost later once you’ve established reliable evaluation criteria. Give each judge only the parts of the trace it needs for its failure mode. Do not give every judge the same full trace by default. Extra context can cause context rot ../../../notes/llm/rag/p6-context rot.html and make the judge worse. Finding the right pieces of context often requires experimentation. Test your choices by comparing the judge’s decisions with human labels. Then, inspect disagreements to see whether the judge lacked necessary evidence or was distracted by irrelevant information. If you’re unsure whether a piece of information helps, try an ablation study. This means removing one piece at a time and checking how the results change against human labels. If performance stays the same or improves, you may be able to leave it out. Long-running agents can produce large traces that fill or exceed the judge’s context window. For these cases, consider giving the judge a tool to search the parts it needs. However, don’t add this unless you absolutely need it, as a tool like this adds additional complexity, cost, and latency. Many applications require a model that can refuse to answer a question when it lacks sufficient information. To evaluate whether this refusal behavior is well-calibrated, you need to test if the model refuses at the appropriate times without refusing to answer questions it should be able to answer. To do this effectively, you should construct an evaluation set that has the following components: While the exact proportion isn’t critical, a balanced set with a roughly equal number of answerable and unanswerable questions is a good starting point. The diversity and difficulty of the questions are more important than the precise ratio. The evaluation itself is a binary Pass/Fail check of the model’s judgment. A “Pass” requires the model to satisfy two conditions: it must answer the answerable questions while also refusing to answer the unanswerable ones. A failure is defined as providing a fabricated answer to an unanswerable question, which indicates poor calibration. In the research literature, this capability is known as “Abstention Ability.” To improve this behavior, it is worth searching for this term on Arxiv https://arxiv.org/search/?query=Abstention+Ability&searchtype=all to understand the latest techniques. For most small to medium-sized companies, appointing a single domain expert as a “benevolent dictator” is the most effective approach. This person becomes the definitive voice on quality standards. The expert might be a psychologist for a mental health chatbot or a lawyer for legal document analysis. A single expert eliminates annotation conflicts and prevents the paralysis that comes from “too many cooks in the kitchen”. The benevolent dictator can incorporate input and feedback from others, but they drive the process. If you feel like you need five subject matter experts to judge a single interaction, it’s a sign your product scope might be too broad. However, larger organizations or those operating across multiple domains like a multinational company with different cultural contexts may need multiple annotators. When you do use multiple people, you’ll need to measure their agreement using metrics like Cohen’s Kappa, which accounts for agreement beyond chance. However, use your judgment. Even in larger companies, a single expert is often enough. Have annotators label the same examples independently before they discuss them. Measure agreement and collect the cases where their labels differ. During an alignment session, ask which part of the rubric caused the disagreement and what rule would make the next decision clear. Update the rubric with a definition, rule, or example that covers the disputed case. Then relabel affected examples. If the annotators still disagree, assign a domain expert to make the final decision and record the reason. Start with a benevolent dictator whenever feasible. Only add complexity when absolutely necessary. Start by scrutinizing your product design. It’s often helpful to surface intermediate outputs users can check before a final result. For example, suppose you have an agent that writes a medical report by synthesizing a patient’s medical history. Instead of asking a doctor to provide feedback on the report, show the extracted facts with links to the source material and let doctors correct a fact or resolve conflicting evidence before generating the report. This also keeps the doctor involved and helps them build trust by checking the work as they go. This is a sketch of how such an interface might look: For more discussion on designing for verification, see “It’s Hard to Eval” Is a Product Smell ../../../blog/posts/eval-smell/index.html . The post expands on this example and discusses several others with before-and-after mockups. After you have designed for verification, make sure the review interface removes friction from reviewing data. See the advice on building a review interface ../../../blog/posts/evals-faq/what-makes-a-good-custom-interface-for-reviewing-llm-outputs.html . Some common tips include: Next, debug the review process. First, try fewer examples so reviewers have time to inspect each one carefully. Have people review the same examples independently and discuss disagreements ../../../blog/posts/evals-faq/how-many-people-should-annotate-my-llm-outputs.html . You can also review examples together to see where people get stuck. Disagreement can reveal unclear instructions or missing information. At the outset, collaborate to establish shared context. Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user expectations, confusing responses, or missing features users expect. As time goes on you should lean towards a benevolent dictator for error analysis: a domain expert or PM who understands user needs. Empower domain experts to evaluate actual outcomes rather than technical implementation. Ask “Has an appointment been made?” not “Did the tool call succeed?” The best way to empower the domain expert is to give them custom annotation tools that display system outcomes alongside traces. Show the confirmation, generated email, or database update that validates goal completion. Keep all context on one screen so non-technical reviewers focus on results. Yes, especially when you’re beginning with evals. I’m often surprised by the number of low-hanging fruit I find while reviewing data that don’t require domain knowledge. For example, I’ve found issues like this in specialized domains as an outsider: Furthermore, ask a domain expert to walk through an example and explain why it is good or bad. Watch what they check and which evidence they need. Use what you learn to build a better annotation interface ../../../blog/posts/evals-faq/what-makes-a-good-custom-interface-for-reviewing-llm-outputs.html that makes reviewing easier. You can also help the team collect interactions and review them regularly. For example, see how product managers and engineers can collaborate on error analysis ../../../blog/posts/evals-faq/should-product-managers-and-engineers-collaborate-on-error-analysis-how.html to get an idea of how to structure cross-functional collaboration. Lastly, make sure you leave judgments that require specialized knowledge to the expert. However, don’t assume you need domain expertise to start being useful Outsourcing error analysis is usually a big mistake with some exceptions . The core of evaluation is building the product intuition that only comes from systematically analyzing your system’s failures. You should be extremely skeptical of this process being delegated. When you outsource annotation, you often break the feedback loop between observing a failure and understanding how to improve the product. Problems with outsourcing include: Instead of outsourcing, focus on building an efficient internal evaluation process. 1. Appoint a “Benevolent Dictator”. For most teams, the most effective strategy is to appoint a single, internal domain expert as the final decision-maker on quality. This individual sets the standard, ensures consistency, and develops a sense of ownership. 2. Use a collaborative workflow for multiple annotators. If multiple annotators are necessary, follow a structured process to ensure alignment: Draft an initial rubric with clear Pass/Fail definitions and examples. Have each annotator label a shared set of traces independently to surface differences in interpretation. Measure Inter-Annotator Agreement IAA using a chance-corrected metric like Cohen’s Kappa. Facilitate alignment sessions to discuss disagreements and refine the rubric. Iterate on this process until agreement is consistently high. Building internal capacity does not mean you have to label every trace. Use these strategies to manage the workload: While outsourcing the core error analysis process is not recommended, there are some scenarios where external help is appropriate: Traces can get large when an agent runs for a long time or retrieves a large amount of context. A useful heuristic is to focus on the first upstream failure ../../../blog/posts/evals-faq/how-do-i-debug-multi-turn-conversation-traces.html . Errors tend to compound, which means you can prioritize earlier ones to save time. Use progressive disclosure in your review tool by showing the most relevant information first and letting reviewers expand details as needed. For example, show the conversation initially, with tool outputs collapsed until a reviewer needs to inspect them. If a single trace is still too large to review, work with the domain expert to identify what they need to check. Build a tool that extracts the relevant evidence and links back to its location in the trace or retrieved document. For example, when reviewing an answer about a long contract, the tool could show the relevant clauses with links to their original pages. Always validate this kind of extraction with a domain expert. Quality is more important than quantity. You can usually learn more from carefully investigating a few failures than from rushing through many traces. LLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. For example, if you let an LLM handle all of error analysis ../../../blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html i.e., reviewing and annotating traces , you might overlook failure cases that matter for your product. Suppose users keep mentioning “lag” in feedback, but the LLM lumps these under generic “performance issues” instead of creating a “latency” category. You’d miss a recurring complaint about slow response times and fail to prioritize a fix. That said, LLMs are valuable tools for accelerating certain parts of the evaluation workflow when used with oversight . In conclusion, start by examining data manually to understand what’s actually going wrong. Use LLMs to scale what you’ve learned, not to avoid looking at data. Automating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When you write a prompt, you are forced to clarify your assumptions and externalize your requirements. Good writing is good thinking