{"slug": "how-to-evaluate-llms-before-production", "title": "How to evaluate LLMs before production", "summary": "GitHub's security team developed a framework for evaluating LLM-based systems in production, focusing on reducing false positives in secret scanning while preserving recall as a safety constraint. The team organized evaluation criteria into primary outcomes, safety constraints, and operational guardrails, and used a decision matrix to advance only experiments that met all thresholds. This approach helped move from prototype to production by prioritizing product decisions over model tweaks.", "body_md": "A language model can perform well on a clean benchmark and still struggle with the cases that matter in production.\n\nBenchmarks and curated datasets are useful when prototyping an LLM-based system. They help teams compare models, test an initial prompt, and determine whether an idea is technically plausible.\n\nBut as a system moves closer to production, the evaluation problem changes.\n\nReal inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated. The evaluation set may not reflect the production distribution. Edge cases that rarely appear in benchmarks can become common sources of failure. Even when offline metrics improve, those results may not translate cleanly into production behavior.\n\nWe encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning.\n\nSecret scanning identifies credentials such as tokens and keys that may have been committed to a repository. Because some candidate strings resemble secrets, but don’t actually represent real credentials, developers may spend time investigating alerts that don’t require remediation.\n\nRather than determine whether an LLM could classify a string correctly, we needed to understand whether the system could reduce noisy alerts while preserving enough recall to remain safe for a security workflow.\n\nIn this post, we share the practices that helped us move from promising prototype results to production. The lessons apply broadly to LLM-powered systems in code analysis, developer tools, security, data analysis, and other production workflows.\n\n1. Start with the product decision, not the model\n\nWhen an LLM system doesn’t perform as expected, the first instinct is often to adjust its technical components.\n\nTeams may rewrite the prompt, add context, introduce another reasoning step, adjust the surrounding pipeline, or switch models. Before making any of these changes, they should define the decision the evaluation is meant to support.\n\nFor our secret-scanning work, we asked:\n\nCan the system reduce false positives while preserving enough recall to be safe in a production security workflow?\n\nTo answer this question, teams must decide which mistakes are acceptable, which metrics should drive the product decision, and which guardrails must remain within their defined thresholds.\n\nIn secret scanning, incorrectly suppressing a real credential can be more consequential than asking a developer to review an additional alert. We therefore did not treat precision and recall as equally interchangeable metrics.\n\nOur primary objective was to reduce false positives and improve precision. Recall served as a safety constraint: an experiment could advance only if any decrease remained within a predefined acceptable range. This gave us a clear way to evaluate tradeoffs. We selected the configuration that achieved the strongest false-positive reduction while satisfying the recall requirement and meeting our operational guardrails.\n\nWe organized the evaluation criteria into three levels:\n\nPrimary outcome\n\nThis measured the user benefit we were trying to improve:\n\nFalse-positive reduction\n\nPrecision\n\nSafety constraint\n\nThis prevented an apparent improvement from introducing unacceptable security risk:\n\nRecall\n\nOperational guardrails\n\nThese determined whether the result was practical to deploy:\n\nLatency\n\nCost\n\nReliability\n\nProduction compatibility\n\nThis distinction prevented us from treating every metric as interchangeable. A change that reduced false positives but significantly lowered recall wasn’t automatically an improvement. Neither was a change that improved quality while making the system too slow, expensive, or difficult to integrate.\n\nConsider two hypothetical experiment results:\n\nExperiment\n\nPrecision\n\nRecall\n\nLatency\n\nDecision\n\nExperiment A\n\nLarge improvement\n\nFalls below the safety guardrail\n\nAcceptable\n\nDon’t advance\n\nExperiment B\n\nModerate improvement\n\nRemains within the guardrail\n\nAcceptable\n\nContinue testing\n\nExperiment A may look stronger if precision is viewed in isolation. Experiment B is more aligned with the product goal because it improves the developer experience without violating the recall guardrail.\n\nBefore evaluating an LLM system, decide what success means for the user and which guardrails the system must respect. We want to generate evidence that supports a product decision.\n\n2. Treat offline evaluation like integration testing\n\nAn LLM-based system continues to change after its first successful evaluation, so evaluation should not be a one-time exercise. Teams revise prompts, adopt new models, change how inputs and context are constructed, and refine the surrounding business logic.\n\nAny of these changes can improve the system, introduce a regression, or shift its behavior in an unexpected way.\n\nFor that reason, we treated offline evaluation similarly to an end-to-end integration test. We reran it whenever we made a meaningful change to the prompt, model, input construction, or broader system logic.\n\nThe evaluation also needed to be repeatable enough that each new result could be compared against a known baseline. For every run, we recorded the prompt, model, dataset version, and system configuration.\n\nThis made it possible to answer questions such as:\n\nDid the new prompt improve precision without reducing recall?\n\nDid the model upgrade help across the dataset or only within certain categories?\n\nDid a change to the input or context fix one error pattern while introducing another?\n\nDid a change to the surrounding logic improve the result consistently, or simply shift where errors appeared?\n\nWithout this discipline, teams can easily compare results generated under different conditions and attribute an improvement to the wrong change.\n\nChange one major variable at a time\n\nRepeatability alone is not enough. Experiments also need to be designed so that the cause of a result is clear.\n\nWe changed one major variable at a time and compared each run against a known baseline. For example, we evaluated a prompt revision separately from a model upgrade before testing the two together.\n\nThis mattered because even small prompt changes could shift model behavior, while a model upgrade could affect quality, cost, latency, or output consistency. If both changed in the same experiment, we would not know which one caused the improvement or regression.\n\nWe treated prompts and evaluation configurations like code. We versioned them, recorded what changed, kept previous configurations reproducible, and made rollback possible.\n\nRun ID\n\nPrompt version\n\nModel version\n\nPrecision\n\nRecall\n\nLatency\n\nNotes\n\nR-001\n\nv1\n\nModel A\n\n0.71\n\n0.78\n\n1.2s\n\nBaseline\n\nR-002\n\nv2\n\nModel A\n\n0.75\n\n0.77\n\n1.2s\n\nPrompt-only change\n\nR-003\n\nv1\n\nModel B\n\n0.74\n\n0.80\n\n1.0s\n\nModel-only change\n\nThe values in the evaluation run tracking table above shown are hypothetical and included only to illustrate how evaluation runs can be tracked and compared.\n\nTest model upgrades regularly\n\nWhen an LLM system underperforms, developers often respond by adding more instructions to the prompt. Sometimes that helps, but not always. For example, the prompt may be carrying complexity that comes from the model itself.\n\nA stronger model may perform better with a simpler prompt than an older model does with extensive tuning. Simpler prompts are also easier to understand, test, and maintain.\n\nModel upgrades still need careful evaluation. A new model may improve performance in one category while introducing regressions elsewhere. It may also affect cost, latency, output formatting, or compatibility with the existing pipeline.\n\nThe evaluation process should be inexpensive and repeatable enough that testing a new model becomes routine. Any meaningful change to the prompt, model, or pipeline should go through offline evaluation before reaching production.\n\n3. Keep offline evaluation close to production\n\nAn offline evaluation is only useful when it resembles the task the system will perform in production.\n\nIn a secret-scanning workflow, the model is rarely evaluating one clean, isolated value. It may need to assess a specific candidate alongside surrounding code and other information that is relevant, incomplete, or potentially distracting. Differences in how that information is presented can materially affect the result.\n\nOur offline evaluation therefore needed to preserve the important characteristics of the production task, including:\n\nThe candidate being evaluated\n\nThe surrounding context available to the model\n\nRelevant supporting information\n\nThe way inputs are formatted and constrained\n\nThe broader system logic around the model\n\nEven small differences can skew the results. A cleaner dataset may exclude ambiguous cases, provide more complete context, or remove nearby values that could distract the model.\n\nConsider a simplified example:\n\n```\nexample_token = \"sample_value_for_documentation\" \nproduction_api_key = get_secret_from_environment() \ncandidate_value = \"flagged_value\"\n```\n\nSuppose candidate_value is the value the system is expected to assess. The model may instead focus on example_token because its variable name appears more security-relevant, producing a plausible explanation about the wrong value.\n\nThis kind of failure is easy to miss when evaluation examples contain only one obvious candidate. It surfaced because the offline evaluation preserved some of the ambiguity and distractions found in real secret-scanning workflows.\n\nThe closer the offline pipeline is to the production pipeline, the more useful the evaluation becomes. When the two differ, a strong offline score may simply reflect an easier problem than the one being deployed.\n\n4. Treat production labels as signals, not unquestionable truth\n\nProduction data can make an evaluation more representative, but its labels often capture workflow outcomes rather than reliable ground truth. A dismissed or resolved secret-scanning alert, for example, does not necessarily represent a false positive.\n\nA developer might resolve an alert because:\n\nThe credential was rotated\n\nThe risk was accepted\n\nThe alert needed to be cleared to unblock a workflow\n\nThe alert was incorrectly classified\n\nThese outcomes may look similar in product data while representing different ground-truth states.\n\nBefore using production labels, ask:\n\nHow was the label created?\n\nDoes it match the question the evaluation is trying to answer?\n\nAre different workflow outcomes being grouped into the same category?\n\nFor important or ambiguous subsets, you may need to complete a manual review. You’re not trying to eliminate every imperfect label, but you need to make sure the evaluation data is accurate enough to support the decision being made.\n\n5. Use synthetic and open datasets to fill coverage gaps\n\nRepresentative production data may be limited, sensitive, or unavailable early in development. Synthetic examples, academic benchmarks, and open datasets can help developers bootstrap an evaluation and expand coverage, but these examples should supplement rather than stand in for production-like data.\n\nWith that in mind, synthetic examples can greatly help fill in the gaps for testing cases that are rare or difficult to collect, such as ambiguous inputs, missing context, unusual formatting, and underrepresented failure patterns. A list of credential strings, for example, can test whether a model recognizes common formats, but it cannot fully evaluate how the model reasons about a candidate within real code.\n\nWe adapted external examples to match our task and reviewed labels that did not align with our product definition. We also used realistic failure patterns to create targeted synthetic cases involving nearby credential-like values, test code, placeholders, indirect references, and missing context.\n\n6. Use error analysis to find what aggregate metrics hide\n\nAggregate metrics tell you whether a system improved overall. Error analysis tells you what to change next.\n\nA higher precision score doesn’t reveal whether the remaining errors come from ambiguous inputs, poor prompt framing, missing context, noisy labels, or a narrow dataset.\n\nTo understand those problems, inspect the failures.\n\nWe reviewed samples of false positives and false negatives and grouped them by their likely source: the model, prompt, input, pipeline, dataset, or label. The recurring issues included several already discussed, such as reasoning about the wrong candidate, missing context, and labels that did not match the evaluation definition.\n\nEach category suggested a different response. Reasoning about the wrong value pointed to prompt or input framing, missing evidence pointed to context construction, and incorrect labels required data cleanup. Repeated domain-specific ambiguity could indicate the need for a clearer product policy or a dedicated evaluation category.\n\nManually reviewing dozens or hundreds of examples takes time, but it often leads to faster progress. Once a recurring failure pattern is clear, the team can make a targeted change and measure whether it solved the problem.\n\nA useful question for each error is: Did this failure come from the model, prompt, input, pipeline, dataset, or label?\n\nThat classification turns a vague quality problem into a concrete engineering task.\n\n7. Use LLM-as-judge to focus human review\n\nReviewing every evaluation example manually may not scale. LLM-as-judge can reduce that burden by classifying clear cases, identifying potentially mislabeled examples, and prioritizing ambiguous cases for human review. Because the judge can also make mistakes or agree with another model for the wrong reason, its output should be treated as another prediction rather than ground truth.\n\nA safer pattern is to use the judge for triage:\n\nAutomatically process clear, low-risk cases.\n\nRoute low-confidence, conflicting, or high-impact cases to human reviewers.\n\nPeriodically sample high-confidence cases to check for systematic errors.\n\nTrack disagreement between the judge, the evaluated system, and human reviewers.\n\nVersion and evaluate the judge prompt like any other model component.\n\nUsed this way, the judge concentrates human attention on the cases where review is most likely to change the outcome.\n\n8. What secret scanning taught us\n\nOur goal was to reduce false positives while preserving recall in a security-sensitive workflow. Offline evaluation gave us a controlled way to compare prompt, model, input, and pipeline changes before beginning online experimentation.\n\nThrough repeated evaluation and targeted error analysis, we reached a 95% reduction in false positives on the evaluated offline dataset while keeping recall within our defined guardrail. More importantly, we understood how the result had been produced: the evaluation reflected the production task more closely, changes were measured against reproducible baselines, and the remaining failure patterns were documented.\n\nOffline evaluation did not prove how the system would behave in every production scenario. It provided enough structured evidence to justify moving to online experimentation with clearly understood risks and guardrails.\n\nChecklist: Before moving an LLM system toward production\n\nUse this checklist to assess whether your evaluation provides enough evidence to move the system forward. Work through each section to confirm that the goals, data, experiments, and remaining production risks are clearly understood.\n\nProduct Goals\n\nIs the product decision and primary success metric clear?\n\nAre the safety and operational guardrails defined?\n\nData and Labels\n\nDoes the evaluation data resemble the production workflow and include difficult cases?\n\nDo we understand how the labels were created and where human review is needed?\n\nEvaluation Rigor\n\nAre the prompt, model, dataset, and pipeline versions recorded?\n\nAre major changes isolated and compared against a known baseline?\n\nError Analysis and Production Readiness\n\nHave false positives and false negatives been reviewed by category?\n\nCan we rerun the evaluation and explain where offline results may differ from production?\n\nEvaluate before you trust\n\nAs LLM-based systems move into production, evaluation should become part of the regular engineering workflow. A strong offline evaluation can show whether the product goal has been met under representative conditions, where uncertainty remains, and whether the system is ready for a controlled production rollout.\n\nProduction uncertainty is unavoidable. Evaluation makes it visible, measurable, and manageable.\n\nMariko is a Principal Applied Scientist at Microsoft, where she leads the development of agentic AI workflows for cybersecurity operations. Her current interests focus on LLM-powered systems, agentic workflows, and applying frontier AI research to real-world products and operations.\n\nZixiao is a Senior Applied Scientist at Microsoft whose work focuses on secret detection and agentic security systems, with an emphasis on translating research advances into practical security capabilities at scale. She also conducts research on LLM fine-tuning, and token-efficient AI systems, with interests in improving the efficiency, scalability, and real-world deployment of foundation models.\n\nChat is great for intent, but agent work gets lost in the scroll. Here is how I use canvases with my agentic workflows—and why your workflow also deserves a canvas.", "url": "https://wpnews.pro/news/how-to-evaluate-llms-before-production", "canonical_source": "https://github.blog/ai-and-ml/llms/how-to-evaluate-llms-before-production/", "published_at": "2026-08-25 21:35:11+00:00", "updated_at": "2026-08-25 21:44:35.993913+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-products", "ai-tools"], "entities": ["GitHub"], "alternates": {"html": "https://wpnews.pro/news/how-to-evaluate-llms-before-production", "markdown": "https://wpnews.pro/news/how-to-evaluate-llms-before-production.md", "text": "https://wpnews.pro/news/how-to-evaluate-llms-before-production.txt", "jsonld": "https://wpnews.pro/news/how-to-evaluate-llms-before-production.jsonld"}}