Offline vs. Online Evals: How to Keep Evaluating Your AI App After It Goes Live A new installment in an LLM evaluation series distinguishes offline evaluation — testing an LLM app on a golden dataset before deployment, which provides a release gate, version comparison and regression testing — from online evaluation, which scores live production traffic with no answer key to catch unexpected user inputs, failures that appear only at scale, and drift. The post outlines building an online eval pipeline in four stages — logging, then sampling and an evaluator, then a dashboard, then alerts — and closes with a self-improving loop connecting offline and online evals. Everything we’ve done so far in this series is offline evaluation : testing an LLM app on a golden dataset before you deploy it. Offline evals give you three big wins: a release gate , version comparison , and regression testing . But once the app is live, three new risks appear that no golden dataset can catch: unexpected user inputs , failures that only show up at scale , and drift . That’s where online evaluation comes in. It evaluates live production traffic, with no answer key . This post covers the difference between the two, the idea of correctness vs. normalcy , and how to build an online eval pipeline step by step: logging → sampling → evaluator → dashboard → alerts . It ends with the self-improving loop that ties offline and online evals together. Here’s what we’ve covered in this series so far: Today’s topic is a very important one: offline evals vs. online evals. First we’ll see what offline evals are, then what online evals are, and then how the two differ. You’ll pick up a lot along the way. Good news: there’s nothing new to learn here. You already know offline evals. Every eval pipeline we built in the previous parts of this series was an offline eval. Here’s the simple definition: An offline eval is any eval you run on your LLM application before you deploy it. Remember the AI exam grader from Part 5 https://pub.towardsai.net/llm-evaluation-105-who-actually-grades-your-llm-programmatic-human-and-llm-as-a-judge-29adf7c1422b ? We built an LLM app that grades written civil services mock-test answers just like a human expert would. We built a golden dataset, used LLM-as-a-Judge as the method, and ran the eval. All of that happened after the software was built, but before it was deployed . That makes it an offline eval. So, in plain words: whenever you have an LLM app and you want to fully test it to see whether it’s ready to deploy, the evals you run are called offline evals . That’s it. We didn’t learn anything new. We just gave a name to what we already know. Why not just build the LLM app and deploy it straight away? Because if you deploy without testing, you have no idea how it will behave in production , and anything can go wrong. Offline evals give you three big benefits . The first and most obvious benefit: you can test the app properly before you release it. But it gets better. In real teams, this whole process build → test → deploy is automated with CI/CD. You can set up a simple gate : Here’s how it works step by step: So the offline eval isn’t just a test. Used properly, it works as a release gate . The second big benefit: offline evals let you compare different versions of your app. Say you’re building the exam grader and you can’t decide between a Claude model and an OpenAI model. Everything else in the code is the same. Only the model is different. Which one gives better results for your use case? Easy. Build two versions of the app and run the same eval on both. Since the eval is the same and the golden dataset is the same, the playing field is level. If Claude scores higher, you go with Claude. You can do this for anything: Whenever you’re stuck between two or more options, your eval tells you which one to pick, by running it on each version of the software. The third benefit is regression testing , a well-known term in software. Regression testing means checking that a change didn’t break something that used to work. Here’s an example. Version 1 of our ed-tech platform’s chatbot is already live, and students are chatting with it. In production, we notice a pattern: whenever students ask about refunds , the chatbot answers in a slightly cold tone. We want it to always talk to students in a warm tone. So we go into the system prompt and add an instruction like: “You have to be very kind and very polite. Answer nicely.” After that change, the chatbot’s personality becomes too soft . Someone asks, “What does the premium plan cost?” The real price is 19,500 , but to sound nice, the chatbot says “It’s around 19,000.” It’s not a perfect example, but it shows the point. In a complex system, when you improve one thing, something else breaks . Performance goes up in one place and drops somewhere else. That’s called a regression . So whenever you improve your app, you have to make sure that: Offline evals help here too. Your golden dataset has all kinds of cases : refund questions, pricing questions, course curriculum questions, and so on. You run the eval on every type and check the results per type. If your success rate on refund questions was 90% before the prompt change, it should stay around 90% after it. If it drops to 80% , that’s a sign of regression, and you shouldn’t ship that change. The idea is simple: whenever you make a small change anywhere the system prompt, the model, the vector database, anything , nothing else should break. These three benefits are why offline evals are so important. You can’t do without them. Say you’ve built your LLM app, run your offline evals, and everything passed. The app is deployed. Now what? Once your app is live, you’ll face three major risks . Take our ed-tech chatbot. When we tested it offline, we built a golden dataset of 200–500 questions that we thought users might ask. We tested the chatbot only on those questions . But when we deploy it, we open it up to the real world. Now students can ask anything , including things we never tested for: In production, your chatbot faces a much bigger superset of inputs than what you tested it on. That’s the first risk. There are small failures you’ll never see in an offline setup . They only show up once the app is deployed. Problems that only happen at scale. Say the platform launches a new course. Suddenly, lots of people arrive at once, and the chatbot has thousands of concurrent users . Latency shoots up. You could never have tested this offline, because you can’t bring thousands of users into an offline test. Subtle bias that only becomes visible across thousands of conversations. Maybe the chatbot is slightly biased against people from non-technical backgrounds . You’ll only spot this after it has chatted with thousands of people. With enough data, a pattern appears: it talks well with people from technical backgrounds, but something is off when it talks to non-technical people. You can’t catch this in advance. Many systematic failures like these can happen in production that you never anticipated. This is the most technical of the three. Drift means your offline eval the one you used to test and approve the deployment slowly becomes obsolete. Here’s how. Today, we built the chatbot with all our current documents: pricing, course pages, curriculum pages, lecture transcripts. Over the year, things change: As a business runs, its documents keep changing. A year later, the documents feeding the RAG chatbot may look very different from the ones we started with. But our test cases and golden dataset were built for today’s data. A year later, the whole data and its distribution have changed, so the golden dataset and the whole eval pipeline become kind of obsolete . The result: when we test offline, the eval pipeline still gives good results . But in production, we see lots of negative feedback from users, because they don’t like how the chatbot behaves. It’s all happening because we keep changing things, but we’re not changing our eval setup. No. And here’s the simple reason. Offline evals need a golden dataset with correct answers , and the evaluation is based on those answers. Once you’re in production, there’s no golden dataset . A user can ask anything, and you don’t have the correct answers. You don’t know which student will ask what tomorrow, or what the right answer to that question is. Conclusion: even after testing with offline evals and deploying, there are many risks in production, and you can’t evaluate them with offline evals. That’s why we need online evals. Here’s the simplest definition: Online evaluation means evaluating your system on live production traffic, after deployment, as real users interact with it. It’s a different type of evaluation that runs in production, after your software is deployed. Its biggest feature: it works without an answer key. It works without a golden dataset. That’s why it’s so critical. It keeps your deployed software running properly, and it tells you when something is going wrong in production. | Aspect | Offline Eval | Online Eval ||-----------------|----------------------------------------|----------------------------------------------|| When | Before deployment | After deployment || Data | Fixed golden dataset you create | Live production traffic || Answer key | Usually yes the golden dataset | No. You estimate quality on the go || Timing | Runs before each release | Runs continuously after deployment || Inputs | Only what you anticipated | Anything can come in || Catches | Regressions | Drift, surprises, emergent bugs || Best used for | Release gating, version comparison, CI | Drift detection, real-world performance || Cost & speed | Fast, cheap, repeatable | Can be costly at scale, so you use sampling | A few notes on this table: The most important thing to understand here: offline and online evals are not rivals. Online evals don’t replace offline evals. They always run together . They’re complementary , and you need both. Here’s a line worth remembering: Offline evals check whether your application works correctly . Online evals check whether your application is running normally in production. That’s the main purpose of each. You can’t use one in place of the other. Let’s make that line concrete with the AI exam grader from Part 5 https://pub.towardsai.net/llm-evaluation-105-who-actually-grades-your-llm-programmatic-human-and-llm-as-a-judge-29adf7c1422b . In the offline eval, what were we trying to do? We wanted to check whether the grader marks answers the same way a human expert does. So correctness there meant: the marks our grader gives should be close to the marks a human gave. We measured that with MAE against human-graded answers. Now say we’re happy with the offline results and we deploy the grader. In production, can we measure whether it’s grading correctly right now? No. Because for the answer the grader is marking right now, we don’t know what a human would have given it. There’s no golden dataset for that answer. No human has graded it. Our system is grading it for the very first time. So in production, we can’t measure correctness. What we can check is: is the system running normally? Here’s a simple way. Store the total score of every paper the grader marks. Then plot the distribution of scores over a one-week window. Say each paper is out of 1,000. In a typical week: You look at week 1, week 2, week 3, and week 4. Every week, the distribution looks about the same. This becomes your baseline : “This is what my grader’s scores normally look like.” From now on, you keep comparing production against this baseline. Then, in week 5 , the graph suddenly looks very different. Most students are now scoring around 800 , and very few are getting 200, 300, or 500. Comparing week by week, you can tell something is different this week . Why is the software suddenly giving such high marks? Is that proof of a bug? Not necessarily. The shift could be due to some other trend. Maybe a batch of very strong students took the test that week. But the point is, you’ve been alerted. Something evaluated the system and told you to look. You go and investigate where the problem is. That’s what an online eval does. It can’t always guarantee correctness sometimes it can, often it can’t . Mostly, it tells you whether your app is running normally. Don’t hold on to this example too tightly. It’s built just to explain the difference between correctness and normalcy . It won’t apply everywhere. Here’s how you could check this in code. A two-sample Kolmogorov–Smirnov test is one simple way to ask “do these two distributions look the same?” : python import numpy as npfrom scipy.stats import ks 2samp Total marks out of 1000 the grader gave, pulled from your logsbaseline scores = np.array load scores weeks= 1, 2, 3, 4 this week scores = np.array load scores weeks= 5 print f"Baseline mean: {baseline scores.mean :.0f}" print f"This week mean: {this week scores.mean :.0f}" stat, p value = ks 2samp baseline scores, this week scores if p value < 0.01: print f"Score distribution has shifted KS={stat:.2f} . Investigate " else: print "Scores look normal compared with the baseline." Good question. There are a few ways. 1. Use metrics that don’t need a correct answer. Take a RAG chatbot. You can measure faithfulness : was the answer generated only from the retrieved context? To check that, you don’t need the correct answer . You have the context, and you have the generated answer. Ask an LLM whether the answer comes from that context, and you get a faithfulness score. 2. When a metric does need a correct answer, find a clever proxy. In the exam grader example, we didn’t have the right marks, so we compared against a baseline distribution instead. Here’s another one. You want to measure your chatbot’s correctness in production. Every question is new, and you don’t have the right answers. So how can you tell if the chatbot is making mistakes? Is there some other signal you can use in place of the answer key? Yes: user feedback. Say your chatbot has a 👍 / 👎 button. If you suddenly see lots of thumbs-down ratings in the last hour, something has gone wrong. The chatbot is probably giving wrong answers. So here, user feedback becomes the stand-in for correctness. This is the most important part of the post. Let’s see what an online evaluation pipeline actually looks like. The first step is logging . Your chatbot is live, and people are talking to it. Before you can evaluate anything, you need to record every conversation that’s happening. If you don’t record it, it’s lost. And if it’s lost, what will you evaluate? The idea behind logging is simple: capture a structured, replayable record of every conversation turn. Say the chatbot is on the platform’s website and has 50,000 conversations a day . You store all 50,000 properly. What exactly do you store? Identifiers Content Operational data Behavioral signals from the user Everything gets stored. Which tool do you use for this? A popular choice is LangSmith . In LangSmith you create projects , and every conversation your app has is stored there as a trace : what the user sent, what your app returned, and lots of metadata. There are a few engineering properties your logging must have: 1. Non-blocking. Logging is an operation too. Your chatbot has to answer, and you also have to log. These two shouldn’t get in each other’s way. Running both together must not increase latency . The conversation happens normally on one side, and logging happens on its own in the background Tracing tools like LangSmith already send traces in the background for you . 2. Durable and queryable. Store logs in something like a data warehouse or an observability tool like LangSmith, in a structured way. The best part: you can fetch them back any time in the future. 3. Late signal attachment. Some user signals don’t come during the conversation. They come later . Say a user chats with the bot, doesn’t get help, and the next day emails your support address to complain. That escalation happened a day after the conversation, but you still need to record it. How? You have the conversation ID , so you link the escalation back to that conversation. 4. PII handling. If a user shares personal, sensitive information in a conversation phone number, address, card number, date of birth, national ID number , you mask it before storing it in LangSmith. That way, no teammate or employee can later pull that information out of your logs. Privacy has to be maintained. Since it’s text, masking is the usual approach PII = Personally Identifiable Information . Regex is just a starting point. Real systems often use a dedicated PII detection library or service. So, step 1: before evaluating anything, log everything. Usually in a data warehouse or an observability tool like LangSmith. Before step 2, we need to know what to pay attention to in production. Which signals tell you whether your chatbot is working properly? There are two kinds: 1. Captured signals: signals that already exist . You just store them. 2. Computed signals: signals you have to calculate or figure out . Just storing the chat doesn’t give you these. You need an evaluator an online evaluator that takes the chat and computes the score. Many captured signals latency, errors, cost are general software metrics that any normal app tracks. There’s nothing LLM-specific about them. | Signal | Type | Needs an evaluator? ||--------------------------------|----------|---------------------|| Thumbs up / down | Captured | No || Latency | Captured | No || Cost per conversation | Captured | No || Token usage | Captured | No || Faithfulness | Computed | Yes || Answer relevance | Computed | Yes || Correctness | Computed | Yes || Hallucination | Computed | Yes || Toxicity | Computed | Yes || Bias and fairness | Computed | Yes | The pipeline is different for each kind. Let’s take a captured signal like latency . There’s nothing to calculate, so after logging, you go straight to the next step: dashboarding . Every production setup has a dashboard that shows these quantities over time : The graph tells you whether the system is working normally. Example: the platform launches a course, and 500 people come to the website at once and start chatting with the bot. Suddenly, the bot’s reply time goes up . The latency graph, flat until now, spikes. That tells you something is wrong. So you quickly add resources . If you’re on AWS, you start more EC2 instances , add more load balancers , and manage the traffic somehow. After a while, latency comes back down. LangSmith has this built in. Under Monitoring , you pick your project and get a ready-made dashboard with charts for trace latency, error rate, LLM call count, LLM latency, and cost . You can choose the time window: last 1 hour, 3 hours, 6 hours , and so on. Here’s an important point: you always look at aggregated numbers. The latency of the current chat doesn’t matter much. In a single conversation, latency can go up and down. What matters is the aggregated latency over the last hour . If the average latency across lots of conversations is rising, something has gone wrong at the system level . The next stage after dashboarding is alerting . Nobody watches graphs all day. Your engineer won’t stare at the dashboard 24 hours a day. They might be on leave, asleep, or busy. So you set up alerts : if a quantity crosses a threshold , an alert is sent on Slack , email , or even WhatsApp less common . For example: “Latency is above 4 seconds.” The engineer jumps in and gets things back to normal. In LangSmith, there’s an Alerts section right next to Monitoring. You can: So dashboarding and alerting come together. To tie it together in LangSmith: a single trace might record a latency of 2.03 seconds . That same number shows up on the Monitoring dashboard, aggregated with all other traces. If the hourly number suddenly rises, the alert fires so you can take action. So for any captured quantity, the flow is: Log → Dashboard → Alert No computation and no evaluator needed. Now the most interesting part: tracking a computed quantity in production. Let’s take hallucination . We want to check, in real time, whether our chatbot is hallucinating. Same as before. If your chatbot has 50,000 conversations a day, log every one of them with a tool like LangSmith. Next, you create an evaluator . And this evaluator is reference-free . Remember reference-based vs. reference-free https://pub.towardsai.net/llm-evaluation-105-who-actually-grades-your-llm-programmatic-human-and-llm-as-a-judge-29adf7c1422b from Part 5? We want the hallucination rate : is the chatbot hallucinating on a given question? Do we have a golden dataset? A correct answer? No. The user asked something, the bot answered, and we have nothing that tells us whether that answer is a hallucination. So this is a reference-free evaluation . But we still need to evaluate. So we use LLM-as-a-Judge . We bring in a more powerful LLM and show it three things: Then we ask: “Is the chatbot hallucinating or not?” We give it a detailed rubric that guides the judge on how to decide. Here’s a big problem. If the chatbot has 50,000 conversations a day , should we run this LLM judge on all 50,000 ? Obviously not. It would be very expensive . You’re already paying to run the chatbot for 50,000 conversations. If you also pay to evaluate every one, your cost could more than double . So there’s an extra step in the middle: sampling . Out of 50,000 conversations a day, you randomly select about 1,000 and run your evaluator the LLM judge only on those 1,000. The evaluator gives you a number: the hallucination rate . You send that number to your dashboard , just like the captured quantities, and from the dashboard you create alerts . So the flow is the same as before, with two extra steps: sampling to save cost and evaluation to compute the metric . Picking 1,000 conversations at random out of 50,000 works. But is it the best strategy? Not really. Not all conversations are the same. Think about it: This is called stratified sampling : The sample you get has a much better chance of catching hallucinations . You can think of the categorizing step as a kind of clustering . To do it, you’ll usually need a small model that detects which conversation has a thumbs down, which one is about refunds, and so on. One caution: because stratified sampling over-represents risky chats, the raw hallucination rate on this sample will be higher than the true rate across all traffic. That’s fine if you use it as an early-warning signal and compare it with its own baseline. If you need the true overall rate, weight each stratum by its real share of traffic. Putting it all together: That’s the entire flow for one online eval pipeline. In practice, you set up multiple evaluators and multiple eval pipelines like this. Here’s how this looks in a real tool. In LangSmith, there’s an Evaluators section, with a “Get started with evaluators” option. Click Show all templates and you’ll see many ready-made evaluators, each one marked LLM-as-a-Judge : For example, the PII leakage evaluator checks whether your chatbot is leaking personal information. Behind the scenes, an LLM-as-a-Judge is doing the work. Click the hallucination template, then: Then you get two options. You can run this evaluator on: What’s the difference? So LangSmith is a complete evaluation platform : you can do both online and offline evaluation there. You can create datasets too. Go to Datasets & Experiments , click New dataset , and keep adding examples. Select that dataset in your evaluator, and it becomes an offline eval. If you have no dataset yet, you’ll see two options: Create a dataset and Run an experiment . Click New experiment , and LangSmith gives you the code to run offline experiments. So in LangSmith you can: Treat this as a starting sketch. Check LangSmith’s current docs for the exact evaluator signatures. Remember the eval workflow from Part 3 https://medium.com/@mominaatherahmed/llm-evaluation-103-how-to-actually-evaluate-an-llm-application-from-start-to-finish-122501bc869b ? Its last step was to deploy and monitor , and we said: when something fails in production, take that case and add it to your offline dataset . We do the same thing here. Say you’re looking at your traces and your team spots a conversation where something went wrong. In LangSmith, there’s an “Add to dataset” button at the top of the trace. Click it, and that conversation becomes part of your offline dataset . The next time you run offline evals, you run them on the updated dataset . You can also add a conversation to an annotation queue , where someone marks what went right and what went wrong. So you keep annotating online data and adding it to your offline data. This forms a circle: This is the self-improving loop . Online failures go into the offline dataset, and once they’re in the offline dataset, they’re part of every future offline eval. That’s how you close the loop . Offline and online evals keep working together, in sync, the whole time. Q: Say my dashboard shows faithfulness 0.87, plus some toxicity and latency numbers. Is my bot good? Every metric needs a baseline , and that baseline usually comes from your offline evaluation . You compare against it. Say your baseline faithfulness is 0.85 . A live score of 0.87 is better than the baseline, so you’re happy. But if, over the last 24 hours , faithfulness suddenly drops to 0.75 , that’s concerning against a 0.85 baseline. An alert fires, or you go in and fix the system. Q: In offline evals, my score went from 92 to 99. Does that mean it’ll be good in production? It depends. Which quantity went from 92 to 99? And did other quantities drop while this one went up? If everything is improving, great. But improving one thing should never make another thing worse . In LLM-based systems, this happens all the time: you improve one thing, you get a regression , and other things drop. That’s exactly what you need to watch for. When you start, you think: “I need to build a chatbot. A RAG chatbot. Get an API key, build it, done.” After learning this, you start thinking differently: “How will my chatbot work properly for thousands, even millions of people?” You’re not just building it anymore. You’re learning to run it properly . That’s what sets you apart, because not many people are learning this yet. | Topic | Key idea ||----------------------------|----------------------------------------------------------------|| Offline eval | Evals run before deployment, on a golden dataset || Offline benefits | Release gate CI/CD , version comparison, regression testing || Production risks | Unanticipated inputs, emergent/systematic failures, drift || Online eval | Evals on live traffic after deployment, no answer key || Offline vs. online | Offline checks CORRECTNESS, online checks NORMALCY || Signals | Captured latency, cost, 👍/👎 vs. computed hallucination... || Captured pipeline | Log → Dashboard → Alert || Computed pipeline | Log → Sample → Evaluator → Dashboard → Alert || Sampling | Stratified beats random: sample more from risky chats || Logging must be | Non-blocking, durable & queryable, late signals, PII masked || Self-improving loop | Production failures → offline dataset → better offline evals | Ask yourself about your own AI project: If the honest answer to any of these is “no” or “not sure,” that’s the gap this post is pointing at. So far, everything in this series has been about application-level evals. Over the next posts, we’ll put all of this into practice: how to build a golden dataset , how to run offline evals , and how to run online evals , using tools like LangSmith , DeepEval , and RAGAS . Next up in this series: Model-level evals and benchmarks: how LLMs themselves are evaluated, before we move back to application evals. If you missed the earlier parts of this series: Offline vs. Online Evals: How to Keep Evaluating Your AI App After It Goes Live https://pub.towardsai.net/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live-efb7fcc3144c was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.