{"slug": "online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions", "title": "Online Evaluation: Building AI Evaluation Pipelines for Real User Interactions", "summary": "Divakar Ungatla's article on Towards Artificial Intelligence introduces online evaluation for AI applications, extending offline evaluation pipelines to assess real user interactions post-deployment. The piece details building a pipeline that captures user interactions, collects explicit feedback, and uses an LLM judge to evaluate response quality, using the Wayfinder flight search application as an example.", "body_md": "*AI Engineering FundamentalsAI Evaluation · Part 6*\n\n← [Part 5](https://medium.com/towards-artificial-intelligence/llm-as-a-judge-building-automated-evaluation-pipelines-for-ai-applications-8680a412a1bd?sharedUserId=divakar.ungatla)\n\nIn the [previous article](https://medium.com/towards-artificial-intelligence/llm-as-a-judge-building-automated-evaluation-pipelines-for-ai-applications-8680a412a1bd?sharedUserId=divakar.ungatla), we explored **LLM-as-a-Judge** and how another large language model can evaluate AI-generated responses using structured evaluation criteria.\n\nThis allows us to evaluate qualities such as helpfulness, relevance, completeness, and groundedness at a much larger scale than relying entirely on human reviewers.\n\n**However, the evaluations we have built so far share an important limitation — they happen offline.**\n\nThey start with evaluation datasets containing scenarios we have prepared in advance. These datasets are extremely useful for testing known behaviors, comparing changes and catching regressions.\n\nBut once an AI application is deployed, users may interact with it in ways we never anticipated.\n\nThey may ask questions that are not present in our evaluation dataset, phrase requests differently, combine constraints in unexpected ways, or expose behaviors that our offline tests never covered.\n\nThis raises an important question:\n\n**How do we evaluate the quality of an AI application when it is being used by real users?**\n\nThis is where **Online Evaluation** comes in.\n\nInstead of evaluating only predefined scenarios from an evaluation dataset, online evaluation evaluates interactions generated while an AI application is actually being used.\n\nThese interactions can provide different kinds of quality signals. Users can directly tell us whether a response was useful, while automated evaluators can assess dimensions such as helpfulness, relevance, completeness, and groundedness across a much larger number of interactions.\n\nAt a high level, online evaluation extends evaluation from the scenarios we prepared **before deployment** to the interactions that actually happen **after deployment**.\n\nIn this article, we’ll build an online evaluation pipeline that captures real user interactions, collects explicit user feedback, and automatically evaluates response quality using an LLM judge.\n\nSo far in this series, we have evaluated AI applications **offline.**\n\nWe start with a predefined evaluation dataset, run the application against those examples and evaluate the generated responses. Because the evaluation cases are known in advance we can run them repeatedly to validate changes and catch regressions.\n\nOnline evaluation starts from a different place - **interactions that actually happen when users use the application. **This difference changes how evaluation works.\n\nIn an offline evaluation dataset, we can define what we expect the application to do. For example, the datasets we built earlier included an **expected_behavior** for each evaluation case.\n\n**A real user interaction does not arrive with an expected_behavior.**\n\nIf a user asks:\n\nShow me flights from Bangalore to Tokyo that arrive before 8 PM.\n\nwe have the user’s query, the application’s response, and potentially the context used to generate it — but nobody has defined the expected behavior for that interaction beforehand.\n\nOnline evaluation therefore needs to derive quality signals from the interaction itself. Two particularly useful sources are:\n\n**User feedback** — what the user tells us about their experience.\n\n**Automated evaluation** — what an evaluator can assess about the quality of the response.\n\nWe’ll implement both.\n\nThroughout this series, we’ve been using [ Wayfinder](https://github.com/DivakarUngatla/wayfinder/tree/v0.5.0), an AI-powered flight search application, to build and explore different evaluation techniques.\n\nWe’ll continue with the same application and extend its evaluation system to support **online evaluation**.\n\nOur implementation will capture real user interactions, collect explicit user feedback, and automatically evaluate response quality using an LLM judge.\n\nThe complete implementation used throughout this article is available in thecompanion GitHub repository.To follow along with exactly the same code shown in this article, check out thev0.5.0release. This ensures the code, commands and screenshots remain consistent over time.\n\nTo follow along locally, clone the repository and check out the release containing the online evaluation implementation:\n\n```\ngit clone https://github.com/DivakarUngatla/wayfinder.gitcd wayfindergit checkout v0.5.0\n```\n\nInstall the project dependencies:\n\n```\nuv sync\n```\n\nBefore running the evaluation, configure the required environment variables.\n\nCreate a .env file:\n\n```\ncp .env.example .env\n```\n\nConfigure the required environment variables:\n\n```\nOPENAI_API_KEY=<your-api-key>LANGSMITH_API_KEY=<your-api-key>LANGSMITH_TRACING=true\n```\n\nTo evaluate an interaction, we first need to capture what happened during it.\n\nFor our flight search application, an interaction gives us three important pieces of information:\n\nUnlike our offline evaluations, these interactions are not coming from a predefined evaluation dataset. They are generated naturally as users interact with the application.\n\nWe’ll use **LangSmith tracing** to capture them.\n\nOur application already has a single [WayfinderAgent.run()](https://github.com/DivakarUngatla/wayfinder/blob/v0.5.0/src/wayfinder/agent/wayfinder_agent.py#L171) entry point, so tracing each interaction requires only a small change:\n\n``` python\nfrom langsmith import traceable@traceable(name=\"WayfinderAgent\")def run(self, user_query: str) -> AgentResponse:    ...\n```\n\nThe @traceable decorator creates a LangSmith trace for every call to the agent, capturing the input and output of the interaction.\n\nWith tracing enabled, let’s run Wayfinder and generate an interaction:\n\n```\nuv run python examples/wayfinder_cli.py\n```\n\nThen ask:\n\n```\nYou: Show flights from Bangalore to Tokyo that arrive before 8 PM\n```\n\nBecause WayfinderAgent.run() is traced, the interaction is automatically captured in LangSmith. Open the configured [LangSmith](https://smith.langchain.com) project to inspect the trace.\n\nThe trace now contains the query, application response, and retrieved flight context we need for evaluation.\n\nMore importantly, **we didn’t have to create an evaluation case beforehand**. The real interaction itself becomes the input to our online evaluation pipeline.\n\nNow that we’re capturing real interactions, the simplest quality signal we can collect is **feedback directly from the user**.\n\nAfter Wayfinder responds, we’ll ask whether the response was helpful:\n\n```\nWas this response helpful? (y/n):\n```\n\nA positive response is recorded as 1 while a negative response is recorded as 0:\n\n```\nscore = 1 if feedback == \"y\" else 0ls_client.create_feedback(    run_id=run_id,    key=\"user_feedback\",    score=score)\n```\n\nThe feedback is attached to the same LangSmith trace that captured the interaction.\n\nLet’s run Wayfinder again:\n\n```\nuv run python examples/wayfinder_cli.py\n```\n\nAfter receiving the response, we can provide feedback directly from the CLI:\n\nOpening the interaction in [LangSmith](https://smith.langchain.com) now shows the user_feedback signal alongside the trace.\n\nA 👍 or 👎 gives us a useful signal about whether the user found the response helpful, but it doesn’t tell us much about the quality of the response itself. Was it relevant? Complete? Grounded in the retrieved context?\n\nAnd many interactions may receive no explicit feedback at all.\n\nTo evaluate response quality more systematically, we need another signal.\n\nAnd many interactions may receive **no explicit feedback at all**.\n\nTo evaluate response quality more systematically, we need another signal.\n\nUser feedback gives us a valuable quality signal, but it is often sparse and doesn’t explain why a response performed poorly.\n\nTo evaluate response quality more systematically, we can use an **LLM as a judge** — another language model that evaluates the application’s response against criteria such as helpfulness, relevance, completeness, and groundedness.\n\nFor online evaluation however, there is an important challenge.\n\nReal user interactions don’t come with a predefined expected_behavior. The evaluator only has what happened during the interaction: the **user’s query, application response and retrieved context**.\n\nOur judge therefore needs to infer the user’s intent from the query and evaluate the response without relying on a predefined reference. This is known as **reference-free evaluation**.\n\nWe’ll create an OnlineLLMJudge for this ( [full code reference](https://github.com/DivakarUngatla/wayfinder/blob/v0.5.0/src/wayfinder/evaluators/online_llm_judge.py#L87)):\n\n``` python\nclass OnlineLLMJudge:    def evaluate(        self,        query: str,        response: str,        context: Any,    ) -> JudgeResult:        ...\n```\n\nThe judge evaluates the response using the same criteria, but its prompt is designed specifically for real interactions. The key instructions are:\n\n```\nImportant: This is a real user interaction, so there is no reference answeror predefined expected behavior available. You must infer the user's intentfrom the original query itself....Instructions:- Infer the user's intent from the original query.- Evaluate the response against each criterion independently using the rubric above.- When evaluating Groundedness, cross-reference factual claims in the response against the supplied context.- Do not assume supporting facts that are not present in the supplied context when assessing Groundedness.\n```\n\nThe result is a structured set of **scores and explanations** for each quality criterion, along with an overall assessment.\n\nThis allows us to automatically evaluate interactions even when the user provides no explicit feedback.\n\nOur OnlineLLMJudge can evaluate an individual interaction, but real applications generate many interactions over time.\n\nWe don’t want to invoke the judge while the user is waiting for a response. Instead, we can evaluate captured interactions separately in the background.\n\nFor Wayfinder, we’ll build a small evaluator job that reads recent interactions from LangSmith ([full code reference](https://github.com/DivakarUngatla/wayfinder/blob/v0.5.0/examples/online_evaluation/evaluate_recent_runs.py#L38)):\n\n```\nruns = ls_client.list_runs(    project_name=project_name,    run_type=\"chain\",    name=\"WayfinderAgent\",    start_time=datetime.now(timezone.utc) - timedelta(days=1),    limit=10)\n```\n\nBecause each trace already contains the interaction data we captured earlier, we can pass it directly to our online judge:\n\n```\nquery = inputs.get(\"user_query\")response = outputs.get(\"response\")flights = outputs.get(\"flights\")result = judge.evaluate(    query=query,    response=response,    context=flights)\n```\n\nOnce the judge evaluates an interaction, we attach its scores and explanations back to the **same LangSmith trace**. We use separate feedback keys for each evaluation criterion so that every quality signal can be inspected independently.\n\n```\nls_client.create_feedback(    run_id=run.id,    key=\"online_judge_helpfulness\",    score=result.helpfulness.score,    comment=result.helpfulness.explanation )\n```\n\nWe use the online_judge_* prefix to distinguish these automated evaluation signals from the user_feedback signal we collected earlier. The same pattern is used for relevance, clarity, completeness, groundedness, instruction following, and the overall score.\n\nWe can run the evaluator independently from Wayfinder:\n\n```\nuv run python examples/online_evaluation/evaluate_recent_runs.py\n```\n\nSince the evaluator may run repeatedly, it also skips interactions that already contain an online_judge_overall result. This prevents previously evaluated traces from being processed again.\n\nThe important architectural point is that **evaluation happens outside the user’s request path**. Wayfinder can respond immediately, while response quality is evaluated separately from the user interaction\n\nIn a production system, this evaluator could run as a **scheduled background job**, periodically processing new interactions. At larger scale, the same pattern could be implemented using asynchronous workers or event-driven pipelines.\n\nWe now have two complementary signals for the same interaction: **explicit user feedback** and **automated evaluation of response quality**.\n\nBecause both are attached to the same LangSmith trace, we can inspect them together. Opening the interaction in [LangSmith](https://smith.langchain.com) now shows the online_judge_* criteria scores alongside user_feedback .\n\nHere, user_feedback captures the user's explicit feedback, while the online_judge_* signals provide structured evaluations across helpfulness, relevance, clarity, completeness, groundedness, and instruction following.\n\nThese signals are independent. User feedback tells us how the user reacted to the response, while the automated judge provides additional evidence about different dimensions of response quality.\n\nPutting everything together, our online evaluation pipeline now looks like this:\n\nLangSmith becomes the central record for each interaction: the application trace captures what happened, while explicit user feedback and automated evaluation add complementary quality signals to the same interaction.\n\nIn this article, we learned how **Online Evaluation** helps us evaluate AI applications using real user interactions rather than only predefined evaluation scenarios.\n\nWe then built an online evaluation pipeline that captures real user interactions, collects explicit user feedback and automatically evaluates response quality using an LLM judge .\n\nTogether, these signals help us understand how our AI application performs beyond predefined evaluation scenarios. By running automated evaluation outside the user’s request path, we can evaluate real interactions without adding latency to the user experience.\n\nEvaluating individual interactions tells us how the application is performing. But when we change a prompt, model, retrieval strategy, or application logic, a new question emerges:\n\n**Did the change actually make the application better?**\n\nIn the next article, we’ll explore **Comparing Evaluation Experiments** — using evaluation results to compare application versions and measure whether changes improve or degrade AI quality.\n\nFollow along as we build a complete AI evaluation toolkit — from evaluation fundamentals to evaluating and improving real-world AI applications.\n\n[Online Evaluation: Building AI Evaluation Pipelines for Real User Interactions](https://pub.towardsai.net/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions-a25081a8f390) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions", "canonical_source": "https://pub.towardsai.net/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions-a25081a8f390?source=rss----98111c9905da---4", "published_at": "2026-08-29 10:01:59+00:00", "updated_at": "2026-08-29 10:18:38.436666+00:00", "lang": "en", "topics": ["ai-products", "ai-tools", "large-language-models", "artificial-intelligence"], "entities": ["Divakar Ungatla", "Towards Artificial Intelligence", "Wayfinder"], "alternates": {"html": "https://wpnews.pro/news/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions", "markdown": "https://wpnews.pro/news/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions.md", "text": "https://wpnews.pro/news/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions.txt", "jsonld": "https://wpnews.pro/news/online-evaluation-building-ai-evaluation-pipelines-for-real-user-interactions.jsonld"}}