Online Evaluation: Building AI Evaluation Pipelines for Real User Interactions

Divakar Ungatla's article on Towards Artificial Intelligence introduces online evaluation for AI applications, extending offline evaluation pipelines to assess real user interactions post-deployment. The piece details building a pipeline that captures user interactions, collects explicit feedback, and uses an LLM judge to evaluate response quality, using the Wayfinder flight search application as an example.

AI Engineering FundamentalsAI Evaluation · Part 6 ← Part 5 https://medium.com/towards-artificial-intelligence/llm-as-a-judge-building-automated-evaluation-pipelines-for-ai-applications-8680a412a1bd?sharedUserId=divakar.ungatla In the previous article https://medium.com/towards-artificial-intelligence/llm-as-a-judge-building-automated-evaluation-pipelines-for-ai-applications-8680a412a1bd?sharedUserId=divakar.ungatla , we explored LLM-as-a-Judge and how another large language model can evaluate AI-generated responses using structured evaluation criteria. This allows us to evaluate qualities such as helpfulness, relevance, completeness, and groundedness at a much larger scale than relying entirely on human reviewers. However, the evaluations we have built so far share an important limitation — they happen offline. They start with evaluation datasets containing scenarios we have prepared in advance. These datasets are extremely useful for testing known behaviors, comparing changes and catching regressions. But once an AI application is deployed, users may interact with it in ways we never anticipated. They may ask questions that are not present in our evaluation dataset, phrase requests differently, combine constraints in unexpected ways, or expose behaviors that our offline tests never covered. This raises an important question: How do we evaluate the quality of an AI application when it is being used by real users? This is where Online Evaluation comes in. Instead of evaluating only predefined scenarios from an evaluation dataset, online evaluation evaluates interactions generated while an AI application is actually being used. These interactions can provide different kinds of quality signals. Users can directly tell us whether a response was useful, while automated evaluators can assess dimensions such as helpfulness, relevance, completeness, and groundedness across a much larger number of interactions. At a high level, online evaluation extends evaluation from the scenarios we prepared before deployment to the interactions that actually happen after deployment . In this article, we’ll build an online evaluation pipeline that captures real user interactions, collects explicit user feedback, and automatically evaluates response quality using an LLM judge. So far in this series, we have evaluated AI applications offline. We start with a predefined evaluation dataset, run the application against those examples and evaluate the generated responses. Because the evaluation cases are known in advance we can run them repeatedly to validate changes and catch regressions. Online evaluation starts from a different place - interactions that actually happen when users use the application. This difference changes how evaluation works. In an offline evaluation dataset, we can define what we expect the application to do. For example, the datasets we built earlier included an expected behavior for each evaluation case. A real user interaction does not arrive with an expected behavior. If a user asks: Show me flights from Bangalore to Tokyo that arrive before 8 PM. we have the user’s query, the application’s response, and potentially the context used to generate it — but nobody has defined the expected behavior for that interaction beforehand. Online evaluation therefore needs to derive quality signals from the interaction itself. Two particularly useful sources are: User feedback — what the user tells us about their experience. Automated evaluation — what an evaluator can assess about the quality of the response. We’ll implement both. Throughout this series, we’ve been using Wayfinder https://github.com/DivakarUngatla/wayfinder/tree/v0.5.0 , an AI-powered flight search application, to build and explore different evaluation techniques. We’ll continue with the same application and extend its evaluation system to support online evaluation . Our implementation will capture real user interactions, collect explicit user feedback, and automatically evaluate response quality using an LLM judge. The complete implementation used throughout this article is available in thecompanion GitHub repository.To follow along with exactly the same code shown in this article, check out thev0.5.0release. This ensures the code, commands and screenshots remain consistent over time. To follow along locally, clone the repository and check out the release containing the online evaluation implementation: git clone https://github.com/DivakarUngatla/wayfinder.gitcd wayfindergit checkout v0.5.0 Install the project dependencies: uv sync Before running the evaluation, configure the required environment variables. Create a .env file: cp .env.example .env Configure the required environment variables: OPENAI API KEY=<your-api-key LANGSMITH API KEY=<your-api-key LANGSMITH TRACING=true To evaluate an interaction, we first need to capture what happened during it. For our flight search application, an interaction gives us three important pieces of information: Unlike our offline evaluations, these interactions are not coming from a predefined evaluation dataset. They are generated naturally as users interact with the application. We’ll use LangSmith tracing to capture them. Our application already has a single WayfinderAgent.run https://github.com/DivakarUngatla/wayfinder/blob/v0.5.0/src/wayfinder/agent/wayfinder agent.py L171 entry point, so tracing each interaction requires only a small change: python from langsmith import traceable@traceable name="WayfinderAgent" def run self, user query: str - AgentResponse: ... The @traceable decorator creates a LangSmith trace for every call to the agent, capturing the input and output of the interaction. With tracing enabled, let’s run Wayfinder and generate an interaction: uv run python examples/wayfinder cli.py Then ask: You: Show flights from Bangalore to Tokyo that arrive before 8 PM Because WayfinderAgent.run is traced, the interaction is automatically captured in LangSmith. Open the configured LangSmith https://smith.langchain.com project to inspect the trace. The trace now contains the query, application response, and retrieved flight context we need for evaluation. More importantly, we didn’t have to create an evaluation case beforehand . The real interaction itself becomes the input to our online evaluation pipeline. Now that we’re capturing real interactions, the simplest quality signal we can collect is feedback directly from the user . After Wayfinder responds, we’ll ask whether the response was helpful: Was this response helpful? y/n : A positive response is recorded as 1 while a negative response is recorded as 0: score = 1 if feedback == "y" else 0ls client.create feedback run id=run id, key="user feedback", score=score The feedback is attached to the same LangSmith trace that captured the interaction. Let’s run Wayfinder again: uv run python examples/wayfinder cli.py After receiving the response, we can provide feedback directly from the CLI: Opening the interaction in LangSmith https://smith.langchain.com now shows the user feedback signal alongside the trace. A 👍 or 👎 gives us a useful signal about whether the user found the response helpful, but it doesn’t tell us much about the quality of the response itself. Was it relevant? Complete? Grounded in the retrieved context? And many interactions may receive no explicit feedback at all. To evaluate response quality more systematically, we need another signal. And many interactions may receive no explicit feedback at all . To evaluate response quality more systematically, we need another signal. User feedback gives us a valuable quality signal, but it is often sparse and doesn’t explain why a response performed poorly. To evaluate response quality more systematically, we can use an LLM as a judge — another language model that evaluates the application’s response against criteria such as helpfulness, relevance, completeness, and groundedness. For online evaluation however, there is an important challenge. Real user interactions don’t come with a predefined expected behavior. The evaluator only has what happened during the interaction: the user’s query, application response and retrieved context . Our judge therefore needs to infer the user’s intent from the query and evaluate the response without relying on a predefined reference. This is known as reference-free evaluation . We’ll create an OnlineLLMJudge for this full code reference https://github.com/DivakarUngatla/wayfinder/blob/v0.5.0/src/wayfinder/evaluators/online llm judge.py L87 : python class OnlineLLMJudge: def evaluate self, query: str, response: str, context: Any, - JudgeResult: ... The judge evaluates the response using the same criteria, but its prompt is designed specifically for real interactions. The key instructions are: Important: This is a real user interaction, so there is no reference answeror predefined expected behavior available. You must infer the user's intentfrom the original query itself....Instructions:- Infer the user's intent from the original query.- Evaluate the response against each criterion independently using the rubric above.- When evaluating Groundedness, cross-reference factual claims in the response against the supplied context.- Do not assume supporting facts that are not present in the supplied context when assessing Groundedness. The result is a structured set of scores and explanations for each quality criterion, along with an overall assessment. This allows us to automatically evaluate interactions even when the user provides no explicit feedback. Our OnlineLLMJudge can evaluate an individual interaction, but real applications generate many interactions over time. We don’t want to invoke the judge while the user is waiting for a response. Instead, we can evaluate captured interactions separately in the background. For Wayfinder, we’ll build a small evaluator job that reads recent interactions from LangSmith full code reference https://github.com/DivakarUngatla/wayfinder/blob/v0.5.0/examples/online evaluation/evaluate recent runs.py L38 : runs = ls client.list runs project name=project name, run type="chain", name="WayfinderAgent", start time=datetime.now timezone.utc - timedelta days=1 , limit=10 Because each trace already contains the interaction data we captured earlier, we can pass it directly to our online judge: query = inputs.get "user query" response = outputs.get "response" flights = outputs.get "flights" result = judge.evaluate query=query, response=response, context=flights Once the judge evaluates an interaction, we attach its scores and explanations back to the same LangSmith trace . We use separate feedback keys for each evaluation criterion so that every quality signal can be inspected independently. ls client.create feedback run id=run.id, key="online judge helpfulness", score=result.helpfulness.score, comment=result.helpfulness.explanation We use the online judge prefix to distinguish these automated evaluation signals from the user feedback signal we collected earlier. The same pattern is used for relevance, clarity, completeness, groundedness, instruction following, and the overall score. We can run the evaluator independently from Wayfinder: uv run python examples/online evaluation/evaluate recent runs.py Since the evaluator may run repeatedly, it also skips interactions that already contain an online judge overall result. This prevents previously evaluated traces from being processed again. The important architectural point is that evaluation happens outside the user’s request path . Wayfinder can respond immediately, while response quality is evaluated separately from the user interaction In a production system, this evaluator could run as a scheduled background job , periodically processing new interactions. At larger scale, the same pattern could be implemented using asynchronous workers or event-driven pipelines. We now have two complementary signals for the same interaction: explicit user feedback and automated evaluation of response quality . Because both are attached to the same LangSmith trace, we can inspect them together. Opening the interaction in LangSmith https://smith.langchain.com now shows the online judge criteria scores alongside user feedback . Here, user feedback captures the user's explicit feedback, while the online judge signals provide structured evaluations across helpfulness, relevance, clarity, completeness, groundedness, and instruction following. These signals are independent. User feedback tells us how the user reacted to the response, while the automated judge provides additional evidence about different dimensions of response quality. Putting everything together, our online evaluation pipeline now looks like this: LangSmith becomes the central record for each interaction: the application trace captures what happened, while explicit user feedback and automated evaluation add complementary quality signals to the same interaction. In this article, we learned how Online Evaluation helps us evaluate AI applications using real user interactions rather than only predefined evaluation scenarios. We then built an online evaluation pipeline that captures real user interactions, collects explicit user feedback and automatically evaluates response quality using an LLM judge . Together, these signals help us understand how our AI application performs beyond predefined evaluation scenarios. By running automated evaluation outside the user’s request path, we can evaluate real interactions without adding latency to the user experience. Evaluating individual interactions tells us how the application is performing. But when we change a prompt, model, retrieval strategy, or application logic, a new question emerges: Did the change actually make the application better? In the next article, we’ll explore Comparing Evaluation Experiments — using evaluation results to compare application versions and measure whether changes improve or degrade AI quality. Follow along as we build a complete AI evaluation toolkit — from evaluation fundamentals to evaluating and improving real-world AI applications. Online Evaluation: Building AI Evaluation Pipelines for Real User Interactions https://pub.towardsai.net/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions-a25081a8f390 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.