# You can't generate evals on autopilot > Source: > Published: 2026-11-14 14:08:05+00:00 Writing good eval criteria is hard. You have to decide what a correct answer looks like and where a reasonable answer stops being good enough. So instead, many of us ask an LLM to write them. Give a capable model your documentation and ask it for questions and rubrics. In minutes, you have hundreds of cases covering knowledge that would have taken you days to encode by hand. As a way to get past the empty page, that's hard to beat. But if you take those generated cases as they are, you've handed the most important decision in your eval to a model: what counts as a good answer. The model bases that decision on what it learned from data available online and on whatever corpus you gave it, and either source can be wrong. Even when the model gets your technology right, it has no way of knowing what someone using that technology day to day would reasonably expect. Consider a criterion a model might generate: Uses the recommended authentication approach. Recommended according to whom? Unless you say otherwise, according to the model's training data. That data reflects what's been written online, which can include outdated samples and [approaches your team stopped recommending years ago](https://developer.microsoft.com/blog/competing-against-yourself/). Grounding the model in your documentation helps, but your documentation can be incomplete or wrong too. Suppose it still recommends an approach your users have moved away from. The generated rubric will require that approach, and every answer that follows it will pass. And whose training data are we talking about? Say you generate the criteria with the most capable model available to you and, to save cost, use a less capable model as the judge. When the generator wrote "recommended," it had a specific approach in mind, but that approach never made it into the criterion. The judge reads the same word and fills it in from its own training data. So the judge scores every answer against its own idea of the recommended approach, which can differ from the one the generator had in mind. Criteria like these look reasonable when you skim them, and producing reasonable-looking text is exactly what language models are good at. If the model generating them doesn't know your technology well, [it can't write criteria that check for correct usage of it](https://developer.microsoft.com/blog/building-ax-evals-that-actually-work/). And it won't tell you which criteria it got wrong, because from where it stands, none of them are. Unchecked cases also make your results hard to interpret. Imagine you give a generated question to the model you actually want to evaluate, and an LLM judge scores its answer against the generated rubric. The judge reports that the answer failed criterion 3. What have you learned? Possibly, that the model under test got the answer wrong. But the question might be ambiguous, or the rubric might require something the question never reasonably asked for. A criterion might depend on information that the rubric doesn't actually contain. Or the source material might not support the expected interpretation at all. Only the first of those explanations says anything about the model you're trying to measure, and until you've checked the eval, you can't rule out the others. You don't need to write every criterion from scratch to avoid this. The model can still help, as long as you use it to find where your judgment is needed. Have a capable model with full access to your corpus generate a small set of reference responses for each case. Start with a clearly correct one and an obviously incorrect one, and add a few subtle failures that look reasonable but violate particular criteria. Then ask it to evaluate those responses against the rubric, one criterion at a time, and compare each verdict with the result that response was designed to produce. You don't need every combination of successes and failures, only enough coverage to show that each criterion can [tell a correct answer from an incorrect one](https://blog.mastykarz.nl/eval-lying/). When a verdict doesn't match, you have to work out why. Is the criterion ambiguous, or does the question lack the context to justify the requirement? Does the rubric assume knowledge it doesn't state? Is the supposedly incorrect response actually defensible, or did the model simply make a mistake? Those are your calls, because each answer depends on what you want the case to test. You can fix or regenerate the criterion. And synthetic cases are cheap enough that throwing a questionable one away is often the most practical option. When every verdict matches, you're not done either. The same model wrote the criteria and the reference responses, then judged them. If it misread a requirement while writing the criterion, it can misread it the same way while judging, and everything will line up. So a clean run only tells you that the eval is consistent with itself and with your corpus. Whether it's consistent with what your users expect is something only you can check. Owning the definition of good doesn't require knowing everything your documentation gets wrong. If you knew where it lagged behind, you'd have fixed it already. When documentation is substantial and has been around for a few years, chances are parts of it are out of date and parts contradict each other, often in places nobody has noticed. And you have no idea what the model was trained on, or what it retained from that training. As whoever owns the product, you should be the expert on your customers: how they use the product and [what they'd reasonably expect from an answer](https://blog.mastykarz.nl/eval-user-success-product-behavior/). Much of that isn't written down anywhere a model could have learned it from. So once you've decided what good looks like, write it into the criterion. List the observable properties an answer must have instead of pointing at "the recommended approach." Otherwise, you've [delegated the decision again](https://blog.mastykarz.nl/eval-judge-know-mean/), this time to the judge that scores your evals. An LLM is a great way to get from an empty page to a first draft of your eval, but the draft is where your work starts. When your dashboard says the model failed 17% of your synthetic evals, you need to know whether those failures mean anything to your users, and no amount of generation can tell you that. You can't generate evals on autopilot.