How to Add Evals to an LLM Feature

A developer explains how to add evals to an LLM feature, using an outbound AI calling agent as an example. The process involves defining a business outcome metric, curating a representative dataset of real test calls, and using the DeepEval framework to automate evaluation in CI. The key takeaway is that evals turn the uncertainty of non-deterministic LLMs into actionable signal for shipping reliable products.

Learning how to add evals to an LLM feature is the difference between shipping a demo and shipping a reliable product. When you embed an LLM into a real feature — a chatbot, a voice agent, a document summarizer — you’re not just calling a model. You’re betting your user’s experience on a non‑deterministic system that can silently break with every prompt tweak, model update, or edge case. That’s why we instrument every LLM feature we build with a purpose‑built eval suite. Here’s how we did it for an outbound AI calling agent https://dev.to/work/ai-calling-agent and how you can do the same. LLMs are non‑deterministic: give them the same input twice, and you’ll get two different responses. That means unit tests that check for exact string matches are useless. As Pragmatic Engineer notes https://newsletter.pragmaticengineer.com/p/evals , you need evals to verify that the solution works well enough — because there’s no guarantee it will. When you’re building a feature that speaks to real customers, like the AI Calling Agent https://dev.to/work/ai-calling-agent dashboard we built, a regression in tone or missed booking intent can cost revenue immediately. Evals turn that uncertainty into signal. We’ll walk through the exact process we followed, from defining success to automating checks in CI, using the DeepEval https://deepeval.com/docs/getting-started framework as an example. You can swap in Evidently AI https://www.evidentlyai.com/llm-guide/llm-evaluation or build your own, but the pattern is the same. Takeaway: Before you pick a metric, write down the one thing that makes the feature “done” — usually a business outcome, not a technical measure. For the AI Calling Agent, the core feature was an outbound call that books a meeting. The success criterion wasn’t “the LLM replied politely.” It was “the agent scheduled a meeting with the right time and date.” This is a reference‑based evaluation: you compare the output to a known ground truth. Evidently AI’s guide https://www.evidentlyai.com/llm-guide/llm-evaluation calls this pattern out as essential for regression testing and experimentation. From that criterion, we derived a concrete metric: successful booking — a boolean that checks whether the transcript contains a confirmed appointment. Later, we layered on softer metrics like tone appropriateness and objection handling . python definition of our primary metric from deepeval.metrics import GEval booking metric = GEval name="successful booking", criteria="Determine whether the agent successfully booked a meeting with the correct date and time.", evaluation steps= "Check if the transcript contains a confirmed appointment.", "Extract the date and time mentioned.", "Verify that the date and time are valid and not contradicted by the user." , evaluation params= LLMTestCaseParams.ACTUAL OUTPUT , Takeaway: Your eval dataset is the spec. If it doesn’t cover the real failures you’ve seen, your eval will pass — and the feature will still fail. We started with 20 transcripts from real test calls: 10 where the agent succeeded, 10 where it failed. For each, we recorded the conversation turn‑by‑turn and the expected outcome. The arXiv practical guide https://arxiv.org/html/2506.13023v1 emphasizes that you should proactively curate representative datasets, not just sample randomly. We included edge cases: strong accents, interruptions, customers who said “call me back later.” We structured the dataset as a list of LLMTestCase objects: python from deepeval.test case import LLMTestCase test cases = LLMTestCase input="Hi, this is Alex from Acme. I'm calling about your interest in the demo...", actual output=" full transcript ", expected output="meeting booked", context= "customer is interested, available Tuesday 2pm" , , ... 19 more cases Takeaway: Mix LLM‑based scoring with deterministic checks. LLM judges are flexible but can drift; non‑LLM scorers ground your eval. For the successful booking metric, we used an LLM judge GPT‑4o with a structured extraction prompt. But we also added a second scorer: a Natural Language Inference NLI model that classifies whether the transcript “entails” the booking. As Confident AI’s metrics guide https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation explains, NLI scorers are a solid non‑LLM option that can be run cheaply and consistently. python from deepeval.metrics import NLIConflictMetric nli metric = NLIConflictMetric threshold=0.5 Evaluate the same case with both scorers booking metric.measure test case nli metric.measure test case We then set a threshold: if both scorers agree, the result is trusted; if they disagree, the case is flagged for manual review. This hybrid approach caught regressions that a single LLM judge missed. Takeaway: An eval that only runs in a notebook is a decoration. The real value comes when it blocks a broken release. We wired the eval suite into a GitHub Action that runs on every pull request to the agent’s prompt configuration. The pipeline fetches the latest model, runs the entire dataset, and fails if the successful booking rate drops below 90% or the NLI score dips. snippet from .github/workflows/evals.yml - name: Run eval suite run: | deepeval test run tests/test booking.py You can also use DeepEval’s ephemeral AI skill https://github.com/confident-ai/deepeval to generate synthetic edge cases and expand your dataset automatically. The point is to make evals as automatic as linting. When we built the AI Calling Agent dashboard https://dev.to/work/ai-calling-agent , evals weren’t an afterthought — they were the first thing we instrumented after the voice pipeline. The agent uses LiveKit for real‑time audio, Twilio for telephony, and OpenAI’s Realtime API. Every tweak to the system prompt or the conversation flow could silently degrade booking rates. We set up a nightly eval job that replays stored transcripts and compares results against the labeled dataset. If a prompt change causes a 2% drop in confirmed bookings, the team gets an alert before a single real call is made. That’s the kind of feedback loop that turns a cool demo into a product ops teams trust. We’ve since applied the same pattern to RAG‑based knowledge bases, chatbots, and internal tools. If you’re shipping an LLM feature today, our AI services https://dev.to/services/ai include eval pipeline design as a first‑class deliverable. Start a project https://dev.to/start and we’ll help you build what you just read — tailored to your stack. Start by defining a single quality criterion for your feature e.g., “the agent booked the appointment” . Then build a small dataset of 10–20 input‑output pairs with ground‑truth labels, pick a scorer like an LLM judge or a classification metric, and run it manually. Once you trust the signal, fold it into CI. LLM‑as‑a‑judge scorers are the most common, but they can be inconsistent. Pair them with non‑LLM scorers like NLI models or structured extraction for stability. The real test is whether your eval catches regressions faster than a user complaint — so run it against known failures. Absolutely. Many eval frameworks use an LLM to judge aspects like correctness, helpfulness, and tone. This is fast and flexible, but it introduces a second layer of non‑determinism. Always validate the judge against a human‑labeled sample before trusting it.