Advanced evals: How to find (and fix) hidden AI failures in your product

Hamel Husain and Shreya Shankar, drawing on work with over 50 AI companies, published an advanced follow-up on building eval systems for AI products, arguing that most teams skip the critical step before writing metrics and end up measuring the wrong things. The post cites Shopify's eval-guided AI workflow builder running 2.2 times faster and 68% cheaper than the frontier-model system it replaced, Cursor cutting costs 41% with its Auto Balance routing evals, Ramp lifting automatic receipt collection accuracy from 35% to 83%, and Harvey nearly doubling its AI contract reviewer's internal quality score. Nearly half of the 25 PM job openings Lenny Rachitsky shared last week ask for eval-writing experience, and Anthropic Labs head Mike Krieger is quoted saying "writing evals is now probably the most important thing" to teach product people.

👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. For more: Lenny’s Jobs https://www.lennysjobs.com/ | Lennybot https://www.lennybot.com/ | Become an AI-Native Builder https://maven.com/tech-for-product/become-an-ai-native-builder and other favorite AI/PM courses https://maven.com/lenny P.S. Get a full free year of Cursor, Notion, Lovable, Replit, Wispr Flow, Linear, Factory, ElevenLabs, PostHog, Granola, Brain.fm, Waking Up, and more, by becoming an Insider subscriber while supplies last . Learn more https://www.lennysproductpass.com/ . Evals have been coming up more and more in my conversations with podcast guests and PMs. And nearly half of the 25 awesome PM https://x.com/lennysan/status/2100657334771237106?s=20 job openings I shared on socials last week ask for experience writing evals. This skill is only becoming more valuable. So I asked Hamel https://www.linkedin.com/in/hamelhusain/ and Shreya https://www.linkedin.com/in/shrshnk/ to write an advanced sequel to their very popular “Building eval systems that improve your AI product” https://www.lennysnewsletter.com/p/building-eval-systems-that-improve post from last year. Drawing from their work with over 50 AI companies, they’ve noticed that most teams jump straight to writing metrics—and end up measuring the wrong things. Below, they share the critical part of the process most teams skip, which steps you can and cannot automate, and a free plugin that lets a coding agent do most of the heavy lifting for you. Enjoy To go deeper, join their upcoming AI Evals for Engineers & PMs course https://maven.com/parlance-labs/evals?promoCode=LENNYSLIST and use the discount code LENNYSLIST at checkout to get 25% off. By now, you’ve probably heard that evals are a defining skill for AI PMs. Mike Krieger, Anthropic’s former CPO and now head of Labs, has said that “if there’s one thing we can teach product people, it’s that writing evals is now probably the most important thing.” Garry Tan, the CEO of Y Combinator, shared that “evals are emerging as the real moat for AI startups.” A number of guests on Lenny’s Podcast have argued that “evals are the new PRDs,” and increasingly, leading companies have been talking about how investing in evals has paid off: - Shopify https://shopify.engineering/fine-tuning-agent-shopify-flow used evals to guide development of an AI workflow builder that was 2.2 times faster and 68% cheaper than the frontier-model system it replaced. - Cursor https://cursor.com/blog/how-cursor-router-works developed its Auto Balance routing performance with evals, resulting in much higher user satisfaction while reducing costs by 41%. - Ramp https://engineering.ramp.com/post/apple-intelligence-receipt-matching increased its automatic receipt collection product’s accuracy from 35% to 83% after investing in evals. - Harvey https://www.harvey.ai/blog/rebuilding-playbook-review-as-a-multi-agent-system rebuilt its AI contract reviewer with evals, nearly doubling the product’s internal quality score. Rippling https://www.rippling.com/blog/building-mcp-server , Glean https://www.glean.com/blog/glean-skills-launch-2026 , Abridge https://tech.abridge.com/blog/offline-evaluations-to-improve-our-systems , ElevenLabs https://elevenlabs.io/blog/eleven-v3-is-now-generally-available , and Robinhood https://robinhood.com/us/en/careers/blog/loop-stays-dumb-so-the-model-can-be-smart/ have also shared how they’ve been using evals to systematically make their AI products better. AI products are easy to change but hard to predict. A prompt, model, or code change can improve one behavior while breaking another. Evals turn your judgment about what “good” looks like into repeatable tests your team can run before shipping. Production errors flagged by evals can become additional test cases that improve your AI, creating an advantage that compounds over time. And now that AI can produce changes faster than people can review them, evals help teams ship quickly by automatically checking if a product still works as intended. In our prior post https://www.lennysnewsletter.com/p/building-eval-systems-that-improve , we laid out the full process for building evals: discover and analyze errors, create customized metrics, and set up a continuous improvement loop. Unfortunately, we’ve found that most teams skip the first stage of error discovery and jump straight to writing metrics. It is easy to see why. Looking through lengthy user session records to find failures feels slow and hard to scale, whereas metrics are concrete and easy to automate. But if you write metrics too early, you end up making too many assumptions about what’s important—and potentially measuring the wrong thing, or the right thing poorly. This is why error discovery is the eval equivalent of product discovery. Just as product discovery shows which problems are worth solving, error discovery reveals which AI failures are worth measuring. Without it, teams risk building dashboards around generic metrics that waste time and steer the product toward the wrong outcomes. We believe error discovery is so important that if you only have time for one part of the eval process, you should prioritize it. In this post, we’ll show you the three steps to running effective error discovery with a coding agent like Codex or Claude. We’ve used this process with more than 50 companies, and each time the tools uncovered major product flaws that were hurting the customer experience. This entire workflow takes only about 30 minutes to complete once you learn the basics. Note: Error discovery has changed a lot since our last post. Our prior post called this process “error analysis.” We now call it “error discovery,” because the goal is to identify failures that are worth measuring. Keep reading to learn about the new approach. Find the errors that matter to your product When you’re building an AI product, you need evals to understand where it makes mistakes. But maintaining evals costs time and money; it doesn’t make sense to measure everything. Good error discovery identifies which failures are worth measuring and tracking over time. Even when you know you need the error discovery stage, it can be tempting to start by handing an agent a folder of traces—complete records of user sessions with your AI product—and asking it to find problems. Agents are often faster than humans at spotting obvious issues and can find patterns we might miss. But they are far less reliable when a failure depends on your definition of a good product experience. You can explain those standards to the agent, but you often only discover them in the first place by reviewing the data. This process, where reviewing examples changes your definition of good, is called criteria drift https://arxiv.org/abs/2404.12272 . For example, below is an interaction from Nurture Boss https://nurtureboss.com/ , an AI leasing assistant we worked with that helps property managers handle conversations with prospective tenants: Prospect: “This is out of my budget. Thank you for your business.” Leasing assistant: “You’re welcome If your situation changes or if you have any other questions in the future, feel free to reach out. Have a great day ” The full conversation from this interaction is provided below in the discussion on traces. To most agents, this looks like a success; they wouldn’t identify an error in this trace. But the product goal is to facilitate sales, which includes finding the right property matches for prospects’ different needs. In this situation, the agent should have explored cheaper units or other properties owned by the same company and offered the prospect alternatives. If we’d prompted the agent to look for this “objection handling” failure up front when we gave it traces to check, the agent would have caught the error automatically. But we’d only know to add “objection handling” to our criteria after seeing this trace ourselves. This is a classic case of criteria drift—and why it’s critical to take a step back to review failures and define success before dispatching an agent to find errors in a stack of traces. In a broader study https://parlance-labs.com/blog/posts/auto-evals/index.html , we ran automated eval tools and coding agents against 100 production traces from this same apartment-leasing assistant. We found the following: 1. Agents missed issues requiring product judgment and context outside the trace such as Markdown formatting in text messages and missed human handoffs in addition to objection handling . 2. Agents are good at catching failures that are obvious inside a trace, like answers that are contradicted by tool output. 3. Agents also find issues that humans miss , but they introduce noise by flagging good responses as failures. Clearly, automated approaches are still useful for finding some types of errors, and they work especially well in conjunction with human judgment. So how do you benefit from an agent’s automation while keeping a human in the loop? The answer is a process that draws on active learning <https://en.wikipedia.org/wiki/Active learning machine learning , a method for choosing the most informative examples to review given limited time. Start with a diverse sample of traces so you cover the range of your data. When you find a failure, look at a few more instances of it before you decide you understand it. After you’ve reviewed enough examples, let the agent annotate traces that you can then accept or reject. All of this bookkeeping and sampling would be difficult to do manually. But coding agents are great at it. The rest of this post walks you through how to work with an agent to do meaningful error discovery step by step, with the aid of an eval skills plugin https://github.com/ai-evals-course/evals-skills we prepared for you. Step 1: Start with your traces Error discovery requires traces. Each trace consists of the user’s input and system prompt, whatever your system did in between retrieval, tool calls, intermediate model calls , and the product’s final output. Each trace should contain enough information for a reviewer to reconstruct what happened and decide whether it was good. You can choose to log this data to a database, eval vendor, or even a local folder. For this post, we will assume a local folder for simplicity. Here is what a trace might look like for the leasing assistant. Note that this is what it looks like in its raw form, and you usually want to render it to be human-readable we will get to that later : If you don’t have traces, you can ask your coding agent to instrument your application so it logs this data. Here’s what a prompt for that might look like: Instrument this app so every user session with the AI is logged as one complete trace. A trace is one user session. Include the user input, the system prompt, every tool call and its result, any retrieved context, every intermediate model call, and the final user-facing output. If this app already sends traces to a vendor LangSmith, Arize, Phoenix, Langfuse, or similar , keep using that. Also write a local copy: one JSON object per session, appended to traces/traces.jsonl. If there is no vendor, the JSONL file is enough. Once your product is instrumented, you’ll need to wait for user activity to collect traces. If you haven’t launched yet, you can try to generate synthetic traces by simulating user queries with an LLM. While synthetic data cannot replace real data, sometimes it’s better than nothing. Pro tip: How to simulate user queries An effective way to simulate user queries is to define a small number of dimensions that you anticipate your product might fail on. For example, dimensions for the leasing assistant might be the task scheduling a tour, asking about pricing, asking about the pet policy , the type of person asking, and whether the request was clear, ambiguous, or out of scope. Then have the model use these dimensions and turn each combination into a natural-language user query. Using dimensions like this helps steer the AI away from producing homogeneous outputs. Once you have defined the dimensions, your coding agent can combine their values into scenarios. Each scenario contains one value from each dimension. The agent can then make a separate model call for each scenario. Here is an example prompt that will generate synthetic data with this approach: Create a script that generates synthetic user queries for an AI leasing assistant. Use the dimensions and values below as fixed inputs. Do not add or change them. Task: scheduling a tour, asking about pricing, asking about the pet policy Renter: first-time renter, relocating family, student Request type: clear, ambiguous, out of scope Create a structured list of test scenarios by combining one value from each dimension. Loop through the scenarios and make a separate model call for each one. Pass only one scenario into each call. Enforce a structured output schema with a single user query field, then save the query alongside its scenario in synthetic queries.jsonl. The prompt above recommends separate model calls for each scenario because we’ve found that asking an agent to one-shot it tends to produce less diversity. Here is an example of what one synthetic user query might look like: { “user query”: “We have two dogs and may be moving next month. Would that work?” } After generating synthetic data, review the examples and remove any that are unrealistic. If you notice scenarios that your dimensions don’t cover, update the dimensions and generate new examples covering those gaps. Producing high-quality synthetic data especially for complex, multi-turn conversations is beyond the scope of this post. We recommend getting real users instead of relying on synthetic data where possible. If you must use synthetic data, we have a skill that can help you with that in the appendix. Step 2: Review and annotate your data Now we are ready to find errors in your product We built an evals plugin to guide you through this process, based on what we learned teaching https://maven.com/parlance-labs/evals?promoCode=LENNYSLIST more than 4,500 PMs and AI engineers and consulting with over 50 companies. Install it with npx skills https://github.com/vercel-labs/skills : npx skills add https://github.com/ai-evals-course/evals-skills https://github.com/ai-evals-course/evals-skills Then point your coding agent at the /evals-start skill or tell it where your data is: “Use the evals-start skill. My traces are in traces/traces.jsonl and I want to find issues occurring in my AI product.” evals-start is the entry point for the evals skills plugin https://github.com/ai-evals-course/evals-skills . It looks at your situation and sends you to the right workflow. In our case, we have traces we haven’t analyzed yet, so it will route us to error-discovery . Customizing the app interface The first thing error-discovery does is read a sample of your records to learn their schema. Then it creates a small review app customized for your data and serves it to you locally. Here’s an example of the app interface that the plugin created for the leasing assistant traces: This interface renders the conversation as a message thread, with tool calls and their outputs displayed alongside. You input your annotations into a free-form text box. Interface customization is an important part of the plugin’s value. Different types of interfaces are best suited to reviewing different types of data. For example, here is what the rendered annotation interface might look like for a writing assistant: In this app, the writing displays as a text block, with flags displayed alongside, which makes it well-suited to reviewing writing. There are several design principles embedded in the skill that guide the way an interface will render. The two most important ones are: 1. Show user-facing output the way the user sees it. For example, an email should look like an email, a PDF should render as a PDF, etc. 2. Expose important metadata that may be helpful for navigation or filtering. In the leasing example, the channel the user comes through is likely important and worth filtering on: SMS, voice, web chat, etc. The skill also instructs your agent to cluster your traces so it can build a diverse initial sample. It mixes cluster representatives with random picks. This approach isn’t perfect, but we’ve found it to be a better starting point than naive approaches to looking at data. Here’s an example of how your coding agent might surface clusters for your review with the writing assistant: In the above example, the annotation app for the writing assistant lets you hover over clusters to review representative documents. If you need to make changes to the clusters, the interface, or anything else, you can just chat with the AI. The biggest advantage of using a coding agent is the ability to change the interface on the fly. If you notice a missing field, want to filter to a slice of traces, or render data differently, all you have to do is ask. For example, if you generated synthetic traces as described in the last section, you can ask your coding agent to create filters based on dimensions you defined like persona or task type. This lets you check whether a failure is concentrated in one kind of scenario. Annotating traces in the app Once the app is running, the skill asks you to review 10 traces before it suggests errors to review. This is a deliberate safeguard against automation bias. Your task is to leave a free-text note on whatever bothers you. Annotations could look like: “The assistant gave up instead of offering alternatives.” “Failed to render a scheduling widget and provided a list of times instead.” “The tone is too formal for this persona.” Some rules of thumb for making annotations: 1. Describe the problem so that a colleague or agent could understand what you meant. “Bad response” is a bad annotation. “The assistant said the unit was available when the tool output showed it was leased” is actionable. 2. Don’t try to do root cause analysis. You’re looking for what went wrong from the user’s perspective, not why your system failed internally. For example, don’t try to diagnose retrieval issues. 3. Stop at the first upstream error. If a trace has multiple problems, annotate the first one you notice. This heuristic helps you save time and focus on the most impactful issues, since upstream failures in agent trajectories tend to be more important than downstream ones. You can relax this constraint later when you are more efficient at this exercise. 4. You only need to annotate failures when starting out. Annotating what makes a trace good might enhance your agent’s understanding but can be skipped if you are short on time. After annotating at least 10 traces, the AI will learn from your initial notes and try to find additional issues in your data that you can accept or reject. It’s important not to blindly accept the agent’s early proposals. Review them carefully and use disagreements to correct the agent’s working understanding. You may need several human-agent iterations before the suggestions become useful. We encourage you to keep annotating beyond the first 10 traces until your own learning has plateaued. As a rule of thumb, aim for 100 traces to make sure you aren’t quitting too early. More traces will also give the AI more signal and improve its suggestions. Here is what the interface might look like as you’re reviewing suggestions for the writing assistant your specific interface will be different, since the AI customizes it for your use case : Here is another view that groups the suggestions by failure mode: If the AI’s suggestions aren’t right, you can tell your coding agent to fix it—just as you can if you want to change the interface. Your annotations do not have to be perfect. The goal is to find actionable issues in your logs rather than perform an exhaustive search. Now that you have set the foundation with hybrid human and AI annotation of your traces, you’re ready to find failure patterns you can act on. Step 3: Turn failure modes into product priorities Once you’ve collected at least 100 diverse annotated traces, your next task is to turn them into a prioritized list of product issues. The AI will attempt to cluster your annotations into failure modes and count them so you can spot patterns. Here is what that looks like for our apartment leasing assistant: