{"slug": "offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live", "title": "Offline vs. Online Evals: How to Keep Evaluating Your AI App After It Goes Live", "summary": "A new installment in an LLM evaluation series distinguishes offline evaluation — testing an LLM app on a golden dataset before deployment, which provides a release gate, version comparison and regression testing — from online evaluation, which scores live production traffic with no answer key to catch unexpected user inputs, failures that appear only at scale, and drift. The post outlines building an online eval pipeline in four stages — logging, then sampling and an evaluator, then a dashboard, then alerts — and closes with a self-improving loop connecting offline and online evals.", "body_md": "Everything we’ve done so far in this series is **offline evaluation**: testing an LLM app on a golden dataset *before* you deploy it. Offline evals give you three big wins: a **release gate**, **version comparison**, and **regression testing**. But once the app is live, three new risks appear that no golden dataset can catch: **unexpected user inputs**, **failures that only show up at scale**, and **drift**. That’s where **online evaluation** comes in. It evaluates live production traffic, with **no answer key**. This post covers the difference between the two, the idea of **correctness vs. normalcy**, and how to build an online eval pipeline step by step: **logging → (sampling → evaluator) → dashboard → alerts**. It ends with the **self-improving loop** that ties offline and online evals together.\n\nHere’s what we’ve covered in this series so far:\n\nToday’s topic is a very important one: **offline evals vs. online evals.** First we’ll see what offline evals are, then what online evals are, and then how the two differ. You’ll pick up a lot along the way.\n\nGood news: there’s nothing new to learn here. **You already know offline evals.** Every eval pipeline we built in the previous parts of this series was an offline eval.\n\nHere’s the simple definition:\n\n***An offline eval is any eval you run on your LLM application* before *you deploy it.***\n\nRemember the AI exam grader from [Part 5](https://pub.towardsai.net/llm-evaluation-105-who-actually-grades-your-llm-programmatic-human-and-llm-as-a-judge-29adf7c1422b)? We built an LLM app that grades written civil services mock-test answers just like a human expert would. We built a golden dataset, used LLM-as-a-Judge as the method, and ran the eval. All of that happened **after the software was built, but before it was deployed**. That makes it an offline eval.\n\nSo, in plain words: whenever you have an LLM app and you want to fully test it to see whether it’s ready to deploy, the evals you run are called **offline evals**. That’s it. We didn’t learn anything new. We just gave a name to what we already know.\n\nWhy not just build the LLM app and deploy it straight away?\n\nBecause if you deploy without testing, you have **no idea how it will behave in production**, and anything can go wrong.\n\nOffline evals give you **three big benefits**.\n\nThe first and most obvious benefit: you can **test the app properly before you release it.**\n\nBut it gets better. In real teams, this whole process (build → test → deploy) is **automated** with CI/CD. You can set up a simple **gate**:\n\nHere’s how it works step by step:\n\nSo the offline eval isn’t just a test. Used properly, it works as a **release gate**.\n\nThe second big benefit: offline evals let you **compare different versions** of your app.\n\nSay you’re building the exam grader and you can’t decide between a **Claude** model and an **OpenAI** model. Everything else in the code is the same. Only the model is different. Which one gives better results for your use case?\n\nEasy. Build **two versions** of the app and run the **same eval** on both. Since the eval is the same and the golden dataset is the same, the playing field is level. If Claude scores higher, you go with Claude.\n\nYou can do this for anything:\n\nWhenever you’re stuck between two or more options, your eval tells you which one to pick, by running it on each version of the software.\n\nThe third benefit is **regression testing**, a well-known term in software.\n\n**Regression testing means checking that a change didn’t break something that used to work.**\n\nHere’s an example. Version 1 of our ed-tech platform’s chatbot is already live, and students are chatting with it. In production, we notice a pattern: whenever students ask about **refunds**, the chatbot answers in a slightly **cold** tone.\n\nWe want it to always talk to students in a warm tone. So we go into the system prompt and add an instruction like: *“You have to be very kind and very polite. Answer nicely.”*\n\nAfter that change, the chatbot’s personality becomes **too soft**. Someone asks, *“What does the premium plan cost?”* The real price is **19,500**, but to sound nice, the chatbot says *“It’s around 19,000.”*\n\nIt’s not a perfect example, but it shows the point. In a complex system, when you improve one thing, **something else breaks**. Performance goes up in one place and drops somewhere else. That’s called a **regression**.\n\nSo whenever you improve your app, you have to make sure that:\n\nOffline evals help here too. Your golden dataset has **all kinds of cases**: refund questions, pricing questions, course curriculum questions, and so on. You run the eval on every type and check the results per type.\n\nIf your success rate on refund questions was **90%** before the prompt change, it should stay **around 90%** after it. If it drops to **80%**, that’s a sign of regression, and you shouldn’t ship that change.\n\nThe idea is simple: **whenever you make a small change anywhere** (the system prompt, the model, the vector database, anything), **nothing else should break.**\n\nThese three benefits are why offline evals are so important. You can’t do without them.\n\nSay you’ve built your LLM app, run your offline evals, and everything passed. The app is deployed. **Now what?**\n\nOnce your app is live, you’ll face **three major risks**.\n\nTake our ed-tech chatbot. When we tested it offline, we built a golden dataset of **200–500 questions** that we thought users might ask. We tested the chatbot **only on those questions**.\n\nBut when we deploy it, we open it up to the real world. Now students can ask **anything**, including things we never tested for:\n\nIn production, your chatbot faces a **much bigger superset** of inputs than what you tested it on. That’s the first risk.\n\nThere are small failures you’ll **never see in an offline setup**. They only show up once the app is deployed.\n\n**Problems that only happen at scale.** Say the platform launches a new course. Suddenly, lots of people arrive at once, and the chatbot has **thousands of concurrent users**. Latency shoots up. You could never have tested this offline, because you can’t bring thousands of users into an offline test.\n\n**Subtle bias that only becomes visible across thousands of conversations.** Maybe the chatbot is slightly **biased against people from non-technical backgrounds**. You’ll only spot this after it has chatted with thousands of people. With enough data, a pattern appears: it talks well with people from technical backgrounds, but something is off when it talks to non-technical people. You can’t catch this in advance.\n\nMany systematic failures like these can happen in production that you never anticipated.\n\nThis is the most technical of the three. **Drift** means your offline eval (the one you used to test and approve the deployment) **slowly becomes obsolete.**\n\nHere’s how. Today, we built the chatbot with all our current documents: pricing, course pages, curriculum pages, lecture transcripts.\n\nOver the year, things change:\n\nAs a business runs, its documents keep changing. A year later, the documents feeding the RAG chatbot may look very different from the ones we started with.\n\nBut our **test cases and golden dataset were built for today’s data.** A year later, the whole data and its distribution have changed, so the golden dataset and the whole eval pipeline become **kind of obsolete**.\n\nThe result: when we test offline, the eval pipeline **still gives good results**. But in production, we see **lots of negative feedback** from users, because they don’t like how the chatbot behaves. It’s all happening because we keep changing things, but **we’re not changing our eval setup.**\n\n**No.** And here’s the simple reason.\n\nOffline evals need a **golden dataset with correct answers**, and the evaluation is based on those answers. Once you’re in production, there’s **no golden dataset**. A user can ask anything, and you don’t have the correct answers. You don’t know which student will ask what tomorrow, or what the right answer to that question is.\n\n**Conclusion:** even after testing with offline evals and deploying, there are many risks in production, and you **can’t evaluate them with offline evals.**\n\nThat’s why we need **online evals.**\n\nHere’s the simplest definition:\n\n***Online evaluation means evaluating your system on live production traffic, after deployment, as real users interact with it.***\n\nIt’s a different type of evaluation that runs in production, **after** your software is deployed.\n\nIts biggest feature: **it works without an answer key.** It works **without a golden dataset.**\n\nThat’s why it’s so critical. It keeps your deployed software running properly, and it tells you when something is going wrong in production.\n\n```\n| Aspect          | Offline Eval                           | Online Eval                                  ||-----------------|----------------------------------------|----------------------------------------------|| When            | Before deployment                      | After deployment                             || Data            | Fixed golden dataset you create        | Live production traffic                      || Answer key      | Usually yes (the golden dataset)       | No. You estimate quality on the go           || Timing          | Runs before each release               | Runs continuously after deployment           || Inputs          | Only what you anticipated              | Anything can come in                         || Catches         | Regressions                            | Drift, surprises, emergent bugs              || Best used for   | Release gating, version comparison, CI | Drift detection, real-world performance      || Cost & speed    | Fast, cheap, repeatable                | Can be costly at scale, so you use sampling  |\n```\n\nA few notes on this table:\n\nThe most important thing to understand here: **offline and online evals are not rivals.** Online evals don’t replace offline evals. They always run **together**. They’re **complementary**, and you need both.\n\nHere’s a line worth remembering:\n\n***Offline evals check whether your application works* correctly*. Online evals check whether your application is running* normally *in production.***\n\nThat’s the main purpose of each. You can’t use one in place of the other.\n\nLet’s make that line concrete with the **AI exam grader** from [Part 5](https://pub.towardsai.net/llm-evaluation-105-who-actually-grades-your-llm-programmatic-human-and-llm-as-a-judge-29adf7c1422b).\n\nIn the offline eval, what were we trying to do? We wanted to check whether the grader **marks answers the same way a human expert does.**\n\nSo **correctness** there meant: *the marks our grader gives should be close to the marks a human gave.* We measured that with MAE against human-graded answers.\n\nNow say we’re happy with the offline results and we deploy the grader. In production, can we measure whether it’s grading **correctly** right now?\n\n**No.** Because for the answer the grader is marking right now, **we don’t know what a human would have given it.** There’s no golden dataset for that answer. No human has graded it. Our system is grading it for the very first time.\n\nSo in production, **we can’t measure correctness.**\n\nWhat we *can* check is: **is the system running normally?**\n\nHere’s a simple way. Store the **total score** of every paper the grader marks. Then plot the **distribution of scores** over a one-week window.\n\nSay each paper is out of 1,000. In a typical week:\n\nYou look at week 1, week 2, week 3, and week 4. Every week, the distribution looks about the same. This becomes your **baseline**: *“This is what my grader’s scores normally look like.”* From now on, you keep comparing production against this baseline.\n\nThen, in **week 5**, the graph suddenly looks very different. Most students are now scoring around **800**, and very few are getting 200, 300, or 500.\n\nComparing week by week, you can tell **something is different this week**. Why is the software suddenly giving such high marks?\n\n**Is that proof of a bug?** Not necessarily. The shift could be due to some other trend. Maybe a batch of very strong students took the test that week. But the point is, **you’ve been alerted.** Something evaluated the system and told you to look. You go and investigate where the problem is.\n\nThat’s what an online eval does. It **can’t always guarantee correctness** (sometimes it can, often it can’t). Mostly, it tells you **whether your app is running normally.**\n\n***Don’t hold on to this example too tightly.*** *It’s built just to explain the difference between* correctness *and* normalcy*. It won’t apply everywhere.*\n\nHere’s how you could check this in code. A two-sample Kolmogorov–Smirnov test is one simple way to ask *“do these two distributions look the same?”*:\n\n``` python\nimport numpy as npfrom scipy.stats import ks_2samp\n# Total marks (out of 1000) the grader gave, pulled from your logsbaseline_scores = np.array(load_scores(weeks=[1, 2, 3, 4]))this_week_scores = np.array(load_scores(weeks=[5]))\nprint(f\"Baseline mean:  {baseline_scores.mean():.0f}\")print(f\"This week mean: {this_week_scores.mean():.0f}\")\nstat, p_value = ks_2samp(baseline_scores, this_week_scores)if p_value < 0.01:    print(f\"Score distribution has shifted (KS={stat:.2f}). Investigate!\")else:    print(\"Scores look normal compared with the baseline.\")\n```\n\nGood question. There are a few ways.\n\n**1. Use metrics that don’t need a correct answer.** Take a RAG chatbot. You can measure **faithfulness**: was the answer generated *only* from the retrieved context? To check that, you **don’t need the correct answer**. You have the context, and you have the generated answer. Ask an LLM whether the answer comes from that context, and you get a faithfulness score.\n\n**2. When a metric does need a correct answer, find a clever proxy.** In the exam grader example, we didn’t have the right marks, so we compared against a **baseline distribution** instead.\n\nHere’s another one. You want to measure your chatbot’s **correctness** in production. Every question is new, and you don’t have the right answers. So how can you tell if the chatbot is making mistakes? Is there some other signal you can use in place of the answer key?\n\nYes: **user feedback.** Say your chatbot has a 👍 / 👎 button. If you suddenly see lots of **thumbs-down ratings** in the last hour, something has gone wrong. The chatbot is probably giving wrong answers. So here, **user feedback becomes the stand-in for correctness.**\n\nThis is the most important part of the post. Let’s see what an online evaluation pipeline actually looks like.\n\nThe first step is **logging**.\n\nYour chatbot is live, and people are talking to it. Before you can evaluate anything, you need to **record every conversation that’s happening.** If you don’t record it, it’s lost. And if it’s lost, what will you evaluate?\n\n***The idea behind logging is simple: capture a structured, replayable record of every conversation turn.***\n\nSay the chatbot is on the platform’s website and has **50,000 conversations a day**. You store all 50,000 properly. What exactly do you store?\n\n**Identifiers**\n\n**Content**\n\n**Operational data**\n\n**Behavioral signals from the user**\n\nEverything gets stored.\n\n**Which tool do you use for this?** A popular choice is **LangSmith**. In LangSmith you create **projects**, and every conversation your app has is stored there as a **trace**: what the user sent, what your app returned, and lots of metadata.\n\nThere are a few engineering properties your logging must have:\n\n**1. Non-blocking.** Logging is an operation too. Your chatbot has to answer, and you also have to log. These two shouldn’t get in each other’s way. Running both together **must not increase latency**. The conversation happens normally on one side, and logging happens on its own in the background *(Tracing tools like LangSmith already send traces in the background for you).*\n\n**2. Durable and queryable.** Store logs in something like a **data warehouse** or an **observability tool** like LangSmith, in a structured way. The best part: you can **fetch them back any time in the future.**\n\n**3. Late signal attachment.** Some user signals don’t come during the conversation. They come **later**. Say a user chats with the bot, doesn’t get help, and the **next day** emails your support address to complain. That escalation happened a day after the conversation, but you still need to record it. How? You have the **conversation ID**, so you link the escalation back to that conversation.\n\n**4. PII handling.** If a user shares **personal, sensitive information** in a conversation (phone number, address, card number, date of birth, national ID number), you **mask it before storing** it in LangSmith. That way, no teammate or employee can later pull that information out of your logs. Privacy has to be maintained. Since it’s text, **masking** is the usual approach (**PII** = Personally Identifiable Information).\n\n*(Regex is just a starting point. Real systems often use a dedicated PII detection library or service.)*\n\nSo, step 1: **before evaluating anything, log everything.** Usually in a data warehouse or an observability tool like LangSmith.\n\nBefore step 2, we need to know **what to pay attention to** in production. Which signals tell you whether your chatbot is working properly?\n\nThere are two kinds:\n\n**1. Captured signals:** signals that **already exist**. You just store them.\n\n**2. Computed signals:** signals you have to **calculate or figure out**. Just storing the chat doesn’t give you these. You need an **evaluator** (an online evaluator) that takes the chat and computes the score.\n\nMany captured signals (latency, errors, cost) are **general software metrics** that any normal app tracks. There’s nothing LLM-specific about them.\n\n```\n| Signal                         | Type     | Needs an evaluator? ||--------------------------------|----------|---------------------|| Thumbs up / down               | Captured | No                  || Latency                        | Captured | No                  || Cost per conversation          | Captured | No                  || Token usage                    | Captured | No                  || Faithfulness                   | Computed | Yes                 || Answer relevance               | Computed | Yes                 || Correctness                    | Computed | Yes                 || Hallucination                  | Computed | Yes                 || Toxicity                       | Computed | Yes                 || Bias and fairness              | Computed | Yes                 |\n```\n\nThe pipeline is different for each kind.\n\nLet’s take a captured signal like **latency**. There’s nothing to calculate, so after logging, you go straight to the next step: **dashboarding**.\n\nEvery production setup has a **dashboard** that shows these quantities **over time**:\n\nThe graph tells you whether the system is working normally.\n\nExample: the platform launches a course, and **500 people** come to the website at once and start chatting with the bot. Suddenly, the bot’s reply time **goes up**. The latency graph, flat until now, spikes. That tells you something is wrong.\n\nSo you quickly **add resources**. If you’re on AWS, you start more **EC2 instances**, add more **load balancers**, and manage the traffic somehow. After a while, latency comes back down.\n\nLangSmith has this built in. Under **Monitoring**, you pick your project and get a ready-made dashboard with charts for **trace latency, error rate, LLM call count, LLM latency, and cost**. You can choose the time window: **last 1 hour, 3 hours, 6 hours**, and so on.\n\nHere’s an important point: **you always look at aggregated numbers.** The latency of the current chat doesn’t matter much. In a single conversation, latency can go up and down. What matters is the **aggregated latency over the last hour**. If the average latency across lots of conversations is rising, something has gone wrong **at the system level**.\n\nThe next stage after dashboarding is **alerting**.\n\nNobody watches graphs all day. Your engineer won’t stare at the dashboard 24 hours a day. They might be on leave, asleep, or busy. So you **set up alerts**: if a quantity **crosses a threshold**, an alert is sent on **Slack**, **email**, or even **WhatsApp** (less common). For example: *“Latency is above 4 seconds.”* The engineer jumps in and gets things back to normal.\n\nIn LangSmith, there’s an **Alerts** section right next to Monitoring. You can:\n\nSo dashboarding and alerting come together.\n\nTo tie it together in LangSmith: a single **trace** might record a latency of **2.03 seconds**. That same number shows up on the **Monitoring** dashboard, aggregated with all other traces. If the hourly number suddenly rises, the **alert** fires so you can take action.\n\nSo for any captured quantity, the flow is:\n\n***Log → Dashboard → Alert***\n\nNo computation and no evaluator needed.\n\nNow the most interesting part: tracking a **computed** quantity in production.\n\nLet’s take **hallucination**. We want to check, in real time, whether our chatbot is hallucinating.\n\nSame as before. If your chatbot has 50,000 conversations a day, log every one of them with a tool like LangSmith.\n\nNext, you create an **evaluator**. And this evaluator is **reference-free**.\n\nRemember [reference-based vs. reference-free](https://pub.towardsai.net/llm-evaluation-105-who-actually-grades-your-llm-programmatic-human-and-llm-as-a-judge-29adf7c1422b) from Part 5?\n\nWe want the **hallucination rate**: is the chatbot hallucinating on a given question? Do we have a golden dataset? A correct answer? No. The user asked something, the bot answered, and we have nothing that tells us whether that answer is a hallucination. So this is a **reference-free evaluation**.\n\nBut we still need to evaluate. So we use **LLM-as-a-Judge**. We bring in a **more powerful LLM** and show it three things:\n\nThen we ask: *“Is the chatbot hallucinating or not?”* We give it a **detailed rubric** that guides the judge on how to decide.\n\nHere’s a big problem. If the chatbot has **50,000 conversations a day**, should we run this LLM judge on **all 50,000**?\n\nObviously not. It would be **very expensive**. You’re already paying to run the chatbot for 50,000 conversations. If you also pay to evaluate every one, your cost could **more than double**.\n\nSo there’s an extra step in the middle: **sampling**. Out of 50,000 conversations a day, you **randomly select about 1,000** and run your evaluator (the LLM judge) only on those 1,000.\n\nThe evaluator gives you a number: the **hallucination rate**. You send that number to your **dashboard**, just like the captured quantities, and from the dashboard you create **alerts**.\n\nSo the flow is the same as before, with two extra steps: **sampling** (to save cost) and **evaluation** (to compute the metric).\n\nPicking 1,000 conversations **at random** out of 50,000 works. But is it the **best** strategy?\n\n**Not really. Not all conversations are the same.** Think about it:\n\nThis is called **stratified sampling**:\n\nThe sample you get has a much better chance of **catching hallucinations**. You can think of the categorizing step as a kind of **clustering**. To do it, you’ll usually need a **small model** that detects which conversation has a thumbs down, which one is about refunds, and so on.\n\n***One caution:*** *because stratified sampling over-represents risky chats, the raw hallucination rate on this sample will be* ***higher*** *than the true rate across all traffic. That’s fine if you use it as an early-warning signal and compare it with its own baseline. If you need the true overall rate, weight each stratum by its real share of traffic.*\n\nPutting it all together:\n\nThat’s the entire flow for **one** online eval pipeline. In practice, you set up **multiple evaluators** and **multiple eval pipelines** like this.\n\nHere’s how this looks in a real tool. In LangSmith, there’s an **Evaluators** section, with a **“Get started with evaluators”** option. Click **Show all templates** and you’ll see many ready-made evaluators, each one marked **LLM-as-a-Judge**:\n\nFor example, the **PII leakage** evaluator checks whether your chatbot is leaking personal information. Behind the scenes, an LLM-as-a-Judge is doing the work.\n\nClick the **hallucination** template, then:\n\nThen you get two options. You can run this evaluator on:\n\nWhat’s the difference?\n\nSo **LangSmith is a complete evaluation platform**: you can do both online and offline evaluation there.\n\nYou can create datasets too. Go to **Datasets & Experiments**, click **New dataset**, and keep adding examples. Select that dataset in your evaluator, and it becomes an offline eval. If you have no dataset yet, you’ll see two options: **Create a dataset** and **Run an experiment**. Click **New experiment**, and LangSmith gives you the code to run offline experiments.\n\nSo in LangSmith you can:\n\n*(Treat this as a starting sketch. Check LangSmith’s current docs for the exact evaluator signatures.)*\n\nRemember the eval workflow from [Part 3](https://medium.com/@mominaatherahmed/llm-evaluation-103-how-to-actually-evaluate-an-llm-application-from-start-to-finish-122501bc869b)? Its last step was to **deploy and monitor**, and we said: when something fails in production, take that case and **add it to your offline dataset**.\n\nWe do the same thing here.\n\nSay you’re looking at your traces and your team spots a conversation where something went wrong. In LangSmith, there’s an **“Add to dataset”** button at the top of the trace. Click it, and that conversation becomes part of your **offline dataset**. The next time you run offline evals, you run them on the **updated dataset**.\n\nYou can also add a conversation to an **annotation queue**, where someone marks what went right and what went wrong. So you keep annotating online data and adding it to your offline data.\n\nThis forms a circle:\n\nThis is the **self-improving loop**. Online failures go into the offline dataset, and once they’re in the offline dataset, they’re part of every future offline eval. That’s how you **close the loop**. Offline and online evals keep working together, in sync, the whole time.\n\n**Q: Say my dashboard shows faithfulness 0.87, plus some toxicity and latency numbers. Is my bot good?**\n\nEvery metric needs a **baseline**, and that baseline usually comes from your **offline evaluation**. You compare against it.\n\nSay your baseline faithfulness is **0.85**. A live score of **0.87** is better than the baseline, so you’re happy. But if, over the last **24 hours**, faithfulness suddenly drops to **0.75**, that’s concerning against a 0.85 baseline. An alert fires, or you go in and fix the system.\n\n**Q: In offline evals, my score went from 92 to 99. Does that mean it’ll be good in production?**\n\nIt depends. **Which** quantity went from 92 to 99? And did **other** quantities drop while this one went up? If everything is improving, great. But improving one thing **should never make another thing worse**. In LLM-based systems, this happens all the time: you improve one thing, you get a **regression**, and other things drop. That’s exactly what you need to watch for.\n\nWhen you start, you think: *“I need to build a chatbot. A RAG chatbot. Get an API key, build it, done.”*\n\nAfter learning this, you start thinking differently: *“How will my chatbot work properly for thousands, even millions of people?”* You’re not just **building** it anymore. You’re learning to **run it properly**. That’s what sets you apart, because not many people are learning this yet.\n\n```\n| Topic                      | Key idea                                                       ||----------------------------|----------------------------------------------------------------|| Offline eval               | Evals run before deployment, on a golden dataset               || Offline benefits           | Release gate (CI/CD), version comparison, regression testing   || Production risks           | Unanticipated inputs, emergent/systematic failures, drift      || Online eval                | Evals on live traffic after deployment, no answer key          || Offline vs. online         | Offline checks CORRECTNESS, online checks NORMALCY             || Signals                    | Captured (latency, cost, 👍/👎) vs. computed (hallucination...) || Captured pipeline          | Log → Dashboard → Alert                                        || Computed pipeline          | Log → Sample → Evaluator → Dashboard → Alert                   || Sampling                   | Stratified beats random: sample more from risky chats          || Logging must be            | Non-blocking, durable & queryable, late signals, PII masked    || Self-improving loop        | Production failures → offline dataset → better offline evals   |\n```\n\nAsk yourself about your own AI project:\n\nIf the honest answer to any of these is “no” or “not sure,” that’s the gap this post is pointing at.\n\nSo far, everything in this series has been about **application-level** evals. Over the next posts, we’ll put all of this into practice: how to **build a golden dataset**, how to **run offline evals**, and how to **run online evals**, using tools like **LangSmith**, **DeepEval**, and **RAGAS**.\n\n**Next up in this series:** Model-level evals and benchmarks: how LLMs themselves are evaluated, before we move back to application evals.\n\n*If you missed the earlier parts of this series:*\n\n[Offline vs. Online Evals: How to Keep Evaluating Your AI App After It Goes Live](https://pub.towardsai.net/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live-efb7fcc3144c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live", "canonical_source": "https://pub.towardsai.net/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live-efb7fcc3144c?source=rss----98111c9905da---4", "published_at": "2026-11-21 12:31:02+00:00", "updated_at": "2026-11-21 12:43:27.260577+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["Claude", "OpenAI"], "also_reported_by": [], "takeaway": "Offline evals on a golden dataset gate releases, compare versions and catch regressions, but live apps need an online eval pipeline — logging, then sampling plus an evaluator, then a dashboard, then alerts — because production traffic has no answer key and surfaces unexpected inputs, scale-only fail", "recommendation": {"advice": "Keep offline evals as a CI release gate on your golden dataset, and add an online pipeline that logs production traffic, samples it through an evaluator, and routes results to a dashboard and alerts to catch drift.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live", "markdown": "https://wpnews.pro/news/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live.md", "text": "https://wpnews.pro/news/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live.txt", "jsonld": "https://wpnews.pro/news/offline-vs-online-evals-how-to-keep-evaluating-your-ai-app-after-it-goes-live.jsonld"}}