# Regression Testing in Agile for RAG Applications > Source: > Published: 2026-10-24 12:01:01+00:00 Here’s a sprint story that’s getting really common. The team building the company’s AI assistant tweaks the chunking strategy, because answers about long policy documents felt a bit thin. Demo day comes, the new answers look great, everyone’s happy. Then a week later support pings: the assistant now gets refund questions wrong. Questions it used to nail. Nobody touched the refund docs. Nobody touched the refund prompt. But retrieval-augmented generation has a lot of moving parts, and changing one of them can quietly shift behavior somewhere else entirely. That’s exactly the kind of thing [**regression testing in agile**](https://keploy.io/blog/community/regression-testing-in-agile) is supposed to catch. The twist is that RAG apps break in ways traditional regression suites were never designed to notice. A RAG app is really a chain of little decisions. Somebody decided how to slice the docs into chunks. Somebody picked an embedding model to turn those chunks into vectors. There’s a retriever deciding which chunks look relevant to a question, and only after all that does the LLM get to write anything. Every single one of those is fair game for a tweak in any given sprint. And they do get tweaked. A lot. Chunk size and overlap. The embedding model. How many chunks get retrieved. A new reranker. A prompt rewrite. A newer LLM version. Fresh documents getting ingested. Any one of those can improve answers on the questions the team was looking at, while making things worse for questions nobody checked. The tricky part is that nothing errors. No stack trace, no 500. The app just answers a little differently, and sometimes a little wrong. It helps to know what you’re hunting for. Most RAG regressions fall into a handful of buckets. The most common is retrieval drift. The right document used to land in the top results and now it doesn’t, so the model answers from the wrong context, or from none at all. Then there’s answer drift, where retrieval’s fine but the model now phrases things in a way that drops a key detail, or hedges where it used to be clear. Faithfulness can slip too, with the model adding claims the retrieved text doesn’t support. And don’t forget the boring ones. A prompt change alters the output format and the frontend can’t parse it anymore. Citations start pointing at the wrong source. The answer gets twice as long and latency creeps up. All very real, all very easy to miss in a demo. Here’s a lesson people tend to learn the hard way. Testing the whole chain with one kind of test is mostly an exercise in frustration. LLM output wobbles between runs, even with the temperature at zero, so end-to-end tests flake and nobody can tell why. You’ll have a much better time testing each layer in its own way. Start here, honestly. It’s the easiest layer to test properly, and it pays off the most. You need a golden set: a bunch of real questions, and for each one, a note of which documents or chunks really ought to come back. Then you just count. How often do those expected sources make it into the top results? That’s basically recall at your top-k cutoff, and it’s a number you can watch sprint over sprint. Retrieval tests are pretty deterministic as long as the index and embedding model don’t change, which makes them great regression checks. And when a team does swap the embedding model or re-chunk everything, comparing these scores before and after shows exactly which questions got worse. This layer’s fuzzier. For each golden question, it helps to write down the key facts a good answer must include, rather than one exact expected answer. Then check whether new answers still include them, and whether they stay faithful to the retrieved context. Plenty of teams use an LLM as a judge here, scoring answers against a rubric. That works, with one caution. The judge is an LLM too, so it wobbles. Pin the judge model and its settings, treat its scores as a trend rather than gospel, and look at the actual answers whenever scores drop. Open-source evaluation libraries like Ragas can take care of a lot of this scoring for you. This one’s plain old software testing, and it’s often skipped. If the frontend expects JSON with an answer field and a citations list, check that shape on every run. If citations should point at real document IDs, check that too. These tests are cheap, they’re deterministic, and they catch a surprising number of “it worked in the demo” bugs. There’s a catch with all of the above. Running live evaluations calls real models and real vector stores, which is slow, costs money and gives slightly different results each time. You don’t want that on every pull request. But a lot of what changes in a sprint isn’t the model at all. It’s the code around it: the orchestration logic, how retrieved chunks get stitched into the prompt, how the response gets parsed, how citations get rendered. That code deserves fast, deterministic tests just like any other. The trick is to record a real run once, model calls and retrieval calls included, and replay those recorded responses in tests. Keploy, an open-source tool, does this by capturing an app’s traffic with eBPF at the kernel level, without code changes, and saving outbound calls as mocks. That covers HTTP calls to LLM APIs, plus queries to vector stores running on Postgres or Redis. Replayed, the pipeline gets identical model and retrieval responses every run, so if a test fails, it’s because the pipeline code changed, not because the model felt creative today. Here’s a rhythm that maps nicely onto how agile teams already work. On every pull request, run the fast, deterministic stuff: contract checks and replayed pipeline tests. They finish in seconds and catch code-level breakage before merge. Nightly or mid-sprint, run the retrieval checks against the golden set, especially if anyone touched chunking, embeddings or the index. Before sprint review, run the full live evaluation, with real retrieval, real model calls and scored answers, and compare everything to the previous sprint’s baseline. Any question whose score dropped gets a look before release. And then feed the golden set. Every time a user reports a bad answer, that question goes in, along with what a good answer needs. Over a few sprints the set grows into a pretty honest picture of what your users actually ask. One habit makes all of this much easier: version every moving part. The prompt templates, the chunking settings, the embedding model, the index build, the LLM version. When a regression shows up, you want to look at a diff and see “this sprint we changed chunk overlap and bumped the model,” not guess. It also makes rollbacks sane. If a change made refund answers worse, you can revert just that part. They split it into layers. Retrieval gets checked against a golden question set with expected sources. Generation gets checked for key facts and faithfulness, often with an LLM judge. And output contracts get checked on every pull request. Fast deterministic tests run per PR, while full live evaluations run before each sprint review. Record real model and retrieval responses once, then replay them in tests. That makes the pipeline code testable on every pull request without live model calls. Tools like Keploy record outbound calls to LLM APIs and vector stores as mocks, so replays return identical responses every run. Re-run retrieval checks on your golden question set and compare the scores with the previous baseline. Any question where the expected source drops out of the top results points to a regression. Then re-run answer evaluations for those questions to see how much it affected users. Here’s how I see it. RAG apps probably need regression testing in agile even more than regular software does. They change in a dozen small ways every sprint, and hardly any of those changes ever throw an error. So you layer it: quick deterministic checks on every PR, live evaluations before each review. That combo lets a team keep making answers better without quietly wrecking the ones that already worked. If your team’s building on RAG right now, start with the golden set. Pick twenty real questions your users ask, write down which documents should answer each one, and check retrieval against that list before your next sprint review. It’s a small effort, and it’ll probably surface at least one question your latest change made worse. [Regression Testing in Agile for RAG Applications](https://pub.towardsai.net/regression-testing-in-agile-for-rag-applications-a8dd827dc3da) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.