{"slug": "llm-as-a-judge-building-llm-based-evaluation-pipelines-for-ai-applications", "title": "LLM-as-a-Judge: Building LLM-Based Evaluation Pipelines for AI Applications", "summary": "LLM-as-a-Judge uses a second large language model to evaluate AI-generated responses against predefined criteria, enabling scalable automated evaluation for AI applications. The approach, demonstrated with the Wayfinder flight search agent, assesses dimensions such as helpfulness, relevance, completeness, groundedness, and instruction following, and is supported by evaluation platforms like LangSmith. The complete implementation is available in the Wayfinder repository on GitHub under the v0.4.0 release.", "body_md": "AI Engineering Fundamentals\n\nAI Evaluation · Part 5\n\n← [Part 4](https://pub.towardsai.net/human-evaluation-building-reusable-evaluation-datasets-for-ai-applications-54f6d93fd2db?sharedUserId=divakar.ungatla)\n\n📦 Complete Code\n\nThe complete implementation used in this article is available in the companionWayfinderrepository on GitHub. To follow along with the exact code and examples from this article, check out thev0.4.0 release.\n\n[View the complete code on GitHub]\n\nIn the previous article, we explored **Human Evaluation** and how human reviewers can assess AI application quality using structured evaluation criteria.\n\nHuman evaluation solves an important problem - it allows us to measure qualities that are difficult to express as deterministic rules, such as helpfulness, relevance, completeness and groundedness.\n\n**However human evaluation has a practical limitation — it does not scale.**\n\nAs AI application grows, the number of interactions increases rapidly. Reviewing every response manually becomes expensive, time-consuming, and difficult to maintain.\n\nThis raises an important question:\n\nCan we automate the evaluation process while maintaining the quality of human judgment ?\n\nThis is where **LLM-as-a-Judge** comes in.\n\nInstead of relying entirely on human reviewers, we can use another large language model to evaluate AI-generated responses against predefined evaluation criteria.\n\nAt a high level, LLM-as-a-Judge introduces another LLM into the evaluation pipeline. The application LLM generates the response, while the judge LLM evaluates that response against predefined criteria.\n\nIn this article, we will explore how LLM-as-a-Judge works, build an evaluation pipeline for an AI application, and see how evaluation platforms like LangSmith help run and track these evaluations at scale.\n\nThe key idea behind LLM-as-a-Judge is simple — instead of asking a language model only to generate responses, we introduce another language model whose responsibility is to evaluate those responses.\n\nIn an AI application, there are now two distinct roles:\n\nThe application LLM tries to solve the user’s request. The judge LLM does not solve the request again; it reviews whether the response produced by the application meets the expected quality standards.\n\nLet’s understand this using [Wayfinder](https://github.com/DivakarUngatla/wayfinder/tree/v0.4.0), our flight search AI application.\n\nA user asks:\n\n```\nFind the cheapest flight from Bangalore to Tokyo tomorrow.\n```\n\nThe [Wayfinder](https://github.com/DivakarUngatla/wayfinder/tree/v0.4.0) agent processes the request, searches available flight data, and generates a response:\n\n```\nThe cheapest flight is Air India AI302 at ₹410.\n```\n\nNow the judge LLM evaluates this response.\n\nThe judge receives:\n\n```\nUser Query:Find the cheapest flight from Bangalore to Tokyo tomorrow.Application Response:The cheapest flight is Air India AI302 at ₹410.Retrieved Context:Available flight information returned by the search tool.Evaluation Criteria:- Helpfulness- Relevance- Completeness- Groundedness- Instruction Following\n```\n\nThe judge is not checking whether the response matches a hardcoded expected answer. Instead, it evaluates whether the response is correct, useful, and supported by the available context.\n\nThe output is an evaluation result:\n\n```\nHelpfulness: 5/5Groundedness: 5/5Instruction Following: 5/5Explanation:The response correctly identifies the cheapest flightand the information is supported by the retrieved flight data.\n```\n\nAn LLM judge is only as effective as the criteria it uses to evaluate responses.\n\nJust like human reviewers need a clear rubric to evaluate AI outputs consistently, an LLM judge also needs predefined evaluation criteria that describe what a good response looks like.\n\nFor Wayfinder, we use the same evaluation dimensions introduced in human evaluation:\n\nThese criteria form the evaluation rubric that defines how the judge LLM should assess response quality.\n\nFor example, instead of asking:\n\n```\nIs this response good?\n```\n\nthe judge receives specific criteria:\n\n```\nEvaluate this response for:- Helpfulness- Groundedness- Instruction Following\n```\n\nThis makes evaluations more consistent and repeatable across different responses.\n\nIn the next section, we will see how these criteria are translated into an LLM-as-a-Judge implementation and how the evaluator produces structured scores and explanations.\n\nNow that we have defined our evaluation criteria, the next step is to build an evaluator that can apply this rubric to AI-generated responses.\n\nAn LLM-as-a-Judge pipeline starts with an evaluation dataset.\n\nThe complete code is here — [Wayfinder](https://github.com/DivakarUngatla/wayfinder/tree/v0.4.0).\n\nAn evaluation dataset contains representative user scenarios that we want our AI application to handle.\n\nEach evaluation sample contains:\n\nFor Wayfinder, our dataset contains scenarios such as:\n\n```\n{  \"inputs\": {    \"query\": \"Find the cheapest flight from Bangalore to Tokyo tomorrow.\"  },  \"outputs\": {    \"expected_behavior\":       \"Recommend the retrieved flight with the lowest price without inventing information.\"  },  \"metadata\": {    \"category\": \"cheapest\"  }}\n```\n\nThe important thing is that the dataset does not contain a fixed expected answer. Instead, it describes the expected behavior of the application.\n\nDuring evaluation, each dataset sample is executed against the AI application.\n\nThe quality of an LLM-as-a-Judge system depends heavily on how the judge prompt is designed.\n\nA good judge prompt should clearly define:\n\nFor Wayfinder, the judge prompt follows this pattern:\n\n```\nYou are an expert evaluator assessing the quality of an AI assistant response.You will be given:- The user's original query.- The expected behavior: a description of what a good assistant response should do for this query.- The assistant's actual response.- The flight data retrieved by the search tool before the response was generated.Your task is to evaluate the assistant's response against each of the following criteria.Evaluation criteria and rubric:{criteria_block}Instructions:- \"Expected behavior\" describes what the application was expected to do for this sample.  It is not a model answer. Use it to understand the intent of the query.- Evaluate the response against each criterion independently.- When evaluating Groundedness, cross-reference the response against the retrieved flight data.- Do not invent facts that are not present in the supplied context.- Score every criterion from 1 to 5 using the rubric above.- Provide a concise explanation for every criterion score.- Provide an overall score from 1 to 5 that reflects your holistic judgment.  Do not calculate the overall score as an average of the criteria scores.- Provide a concise overall explanation.\"\"\"User Query:{query}Application Response:{response}Retrieved Context:{context}Provide a score from 1-5 for each criterionand explain your reasoning.\n```\n\nThe judge is not asked to generate a better response. Its only responsibility is to evaluate the response it receives.\n\nFree-form evaluation responses from the judge are difficult to analyze automatically.\n\nInstead, the evaluator asks the LLM Judge to return structured output. In our implementation, we enforce a predefined response schema so every evaluation produces a consistent format.\n\nThe schema contains:\n\nExample:\n\n```\n{  \"criteria_scores\": [    {      \"criterion\": \"Helpfulness\",      \"score\": 5,      \"explanation\": \"The response directly answers the user's request.\"    },    {      \"criterion\": \"Groundedness\",      \"score\": 5,      \"explanation\": \"The response is supported by the provided context.\"    },    {      \"criterion\": \"Instruction Following\",      \"score\": 5,      \"explanation\": \"The response satisfies the user's requirements.\"    }  ],  \"overall_score\": 5}\n```\n\nWith the evaluation dataset, criteria, judge prompt, and structured output format in place, we can now implement the LLM-as-a-Judge evaluator.\n\nThe evaluator acts as a reusable component that takes an AI application’s output and evaluates it using the predefined rubric.\n\nAt a high level, the judge performs four steps:\n\nThe evaluator receives:\n\n```\nUser QueryApplication ResponseExpected BehaviorRetrieved ContextEvaluation Criteria\n```\n\nand produces:\n\n```\nCriterion Scores+Explanations+Overall Assessment\n```\n\nThe application code exposes a simple interface:\n\n```\nresult = judge.evaluate(    query=query,    expected_behavior=expected_behavior,    response=response,    retrieved_flights=retrieved_context)\n```\n\nInternally, the evaluator builds the judge prompt:\n\n```\nmessages = [    {        \"role\": \"system\",        \"content\": judge_instructions    },    {        \"role\": \"user\",        \"content\": evaluation_input    }]\n```\n\nThe system prompt defines the judge’s role, evaluation criteria and scoring guidelines. The user message provides the specific evaluation sample, including the query, response and supporting context.\n\nThe LLM response is then parsed into the structured evaluation model:\n\n```\nJudgeResult(    criteria_scores=[        CriterionScore(            criterion=\"Groundedness\",            score=5,            explanation=\"The response is supported by the retrieved context.\"        )    ],    overall_score=5,    overall_explanation=\"The response satisfies the user's request.\")\n```\n\nKeeping the judge as a separate component makes it reusable. The same evaluator can be applied to different versions of an AI application using the same evaluation dataset and criteria.\n\nThis allows teams to measure whether changes actually improve the system:\n\nInstead of manually reviewing responses after every change, teams can run repeatable evaluations and compare results across versions.\n\nThe complete implementation used throughout this article is available in the[companion GitHub repository].To follow along with exactly the same code shown in this article, check out thev0.4.0release.This ensures the code, commands, and screenshots remain consistent over time.\n\nTo run the examples locally, clone the repository and checkout the release containing the LLM-as-a-Judge implementation.\n\n```\ngit clone https://github.com/DivakarUngatla/wayfinder.gitcd wayfindergit checkout v0.4.0-llm-judge\n```\n\nInstall the project dependencies:\n\n```\nuv sync\n```\n\nBefore running the evaluation, configure the required environment variables.\n\nCreate a .env file:\n\n```\ncp .env.example .env\n```\n\nAdd your OpenAI API key:\n\n```\nOPENAI_API_KEY=<your-api-key>\n```\n\nFor LangSmith experiments later in this article, also configure:\n\n```\nLANGSMITH_API_KEY=<your-api-key>LANGSMITH_TRACING=true\n```\n\nThe LLM-as-a-Judge evaluator uses the evaluation dataset created earlier. The complete dataset containing evaluation scenarios, expected behavior and metadata is available in the companion repository:\n\n[examples/llm_judge_evaluation/llm_judge_evaluation_dataset.jsonl](https://github.com/DivakarUngatla/wayfinder/blob/v0.4.0/examples/llm_as_a_judge/wayfinder_llm_judge_evaluation_v1.jsonl)\n\nThe evaluator loads each sample from this dataset, runs the application, and passes the generated response to the LLM judge for evaluation.\n\n```\nuv run examples/llm_judge_evaluation/local_evaluation.py\n```\n\nFor each evaluation sample, the judge evaluates the generated response against the predefined criteria and returns structured scores along with explanations.\n\nFor example, consider the arrival-time constraint scenario:\n\n```\nUser Query:Which flight from Bangalore to Tokyo arrives before 8 PM?\n```\n\nThe judge evaluates the response across criteria that is provided to it.\n\nAfter evaluating all samples in the dataset, the pipeline generates an overall evaluation summary.\n\nWhile this run completed successfully, evaluation becomes especially valuable when it detects regressions after application changes.\n\nAfter establishing the baseline evaluation, we can introduce a controlled change to simulate a regression.\n\nIn a real AI system, changes to prompts, models or retrieval logic can unintentionally affect response quality.\n\nFor this example, let us temporarily modify the response-generation prompt. Change the below line in the\n\n```\nIf the user asks for cheapest, select the flight with the lowest price among flights satisfying all constraints.\n```\n\nto\n\n```\nIf the user asks for cheapest, provide the two cheapest available options.\n```\n\nThe evaluation dataset remains unchanged. Only the application behavior changes.\n\nRunning the evaluation again produces something as shown below\n\nThe LLM judge identifies that the response is technically correct but does not fully satisfy the user’s intent.\n\nThe response identifies the cheapest flight correctly, but it provides an additional flight even though the user requested only the cheapest option.\n\nThe evaluator detects this through criteria such as:\n\nThis demonstrates an important advantage of LLM-as-a-Judge: it can evaluate whether a response satisfies the user’s intent, not just whether the output contains valid data.\n\nOnce we start running evaluations repeatedly across different application versions, we need a way to track experiments and compare results. Evaluation platforms like LangSmith provide the required infrastructure.\n\nThe same evaluation dataset used for local evaluation can be uploaded to LangSmith.\n\nThe evaluation dataset contains:\n\nThe dataset used in this article is available in the companion repository.\n\nwayfinder_llm_judge_evaluation_v1.jsonl\n\nCreate a new dataset and upload the JSONL file.\n\nLangSmith automatically detects the fields from the dataset.\n\nFor Wayfinder:\n\nThe LangSmith evaluation runner uses the same LLM-as-a-Judge evaluator that we built earlier.\n\nRun:\n\n```\nuv run examples/llm_as_a_judge/langsmith_evaluation.py\n```\n\nThe evaluator executes each dataset example against Wayfinder and sends the generated response to the judge.\n\nOnce the evaluation completes, click on the langsmith link displayed to view the results\n\nFor each example, we can inspect:\n\nThe important point is that the evaluator itself does not change. The same LLM-as-a-Judge component can run locally or through LangSmith.\n\nIn this article, we explored **LLM-as-a-Judge**, an evaluation approach where an LLM acts as an evaluator to assess AI-generated responses against predefined criteria.\n\nWe built an end-to-end LLM-as-a-Judge evaluation pipeline for an AI-powered flight search application, covering evaluation criteria, judge prompts, structured evaluation results, and running evaluations locally and through LangSmith.\n\nUnlike traditional rule-based checks, LLM-as-a-Judge can evaluate whether AI responses satisfy user intent while considering the context available to the application. It provides both scores and explanations, helping teams understand and improve AI application quality.\n\nLLM-as-a-Judge enables powerful offline evaluation using curated scenarios. However, offline evaluation only covers predefined cases. Production AI systems also need evaluation approaches that learn from real user interactions and live application behavior.\n\nIn the next article, we’ll explore **Online Evaluation**, where we’ll learn how to evaluate AI applications using real-world usage data and feedback signals.\n\nFollow along as we build a complete AI evaluation toolkit — from deterministic rule-based checks to human evaluation, LLM-as-a-Judge, evaluation datasets, and production-scale evaluation workflows.\n\n[LLM-as-a-Judge: Building LLM-Based Evaluation Pipelines for AI Applications](https://pub.towardsai.net/llm-as-a-judge-building-automated-evaluation-pipelines-for-ai-applications-8680a412a1bd) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/llm-as-a-judge-building-llm-based-evaluation-pipelines-for-ai-applications", "canonical_source": "https://pub.towardsai.net/llm-as-a-judge-building-automated-evaluation-pipelines-for-ai-applications-8680a412a1bd?source=rss----98111c9905da---4", "published_at": "2026-08-22 04:10:17+00:00", "updated_at": "2026-08-22 04:42:45.290013+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models"], "entities": ["Wayfinder", "LangSmith", "GitHub", "Divakar Ungatla"], "alternates": {"html": "https://wpnews.pro/news/llm-as-a-judge-building-llm-based-evaluation-pipelines-for-ai-applications", "markdown": "https://wpnews.pro/news/llm-as-a-judge-building-llm-based-evaluation-pipelines-for-ai-applications.md", "text": "https://wpnews.pro/news/llm-as-a-judge-building-llm-based-evaluation-pipelines-for-ai-applications.txt", "jsonld": "https://wpnews.pro/news/llm-as-a-judge-building-llm-based-evaluation-pipelines-for-ai-applications.jsonld"}}