Why One RAG Wasn't Enough: Building a Multi-RAG Pipeline for Jira Backlog Analysis

A developer building a Jira Backlog Analyzer found that a single Retrieval-Augmented Generation (RAG) pipeline produced generic or outdated recommendations. By splitting project knowledge into three separate RAG sources—release notes, program context, and roadmap information—the system could generate more accurate and context-aware summaries and suggestions.

Large language models are only as good as the context you give them. While building an LLM-powered Jira Backlog Analyzer, I learned that simply adding Retrieval-Augmented Generation RAG wasn't enough. The real breakthrough came from treating different types of project knowledge as different kinds of organizational memory. One of the goals of my Jira Backlog Analyzer https://dev.to/jubilee happy a567008f769/how-i-auto-groomed-500-jira-tickets-with-ml-and-llm-1j7k was pretty simple: help project managers make sense of hundreds of backlog items. The analyzer groups related Jira tickets, flags potential duplicates, and generates executive summaries covering technical trends and priorities. The first prototype worked better than I expected. Given nothing but the raw Jira tickets, the LLM could summarize clusters, spot duplicate issues, and even suggest what to do next. Except a lot of those suggestions were generic to the point of being useless. Take one cluster of tickets about slow API response times. The model recommended: "Consider optimizing query performance to improve response times." Sounds reasonable. Except a caching layer that had already cut those response times in half had shipped two releases earlier. The recommendation wasn't wrong exactly, it was just out of date. That kept happening. The model would suggest prioritizing security work without knowing the current program increment was already focused on reliability. The language always sounded confident. The substance often wasn't there. It read well and helped with almost nothing. So I did what most people do at this point: I added RAG. It helped, but only partway. Most RAG tutorials assume you have one pile of documents. You retrieve the most relevant chunks, stuff them into the prompt, and let the model do the rest. That works fine for a lot of question-answering use cases. Software projects are messier than that. Not all project knowledge is trying to answer the same question. Once I actually sat down and thought about what a project manager needs to know, it broke into three buckets: Those are three very different questions, and no single document collection answers all of them well. So instead of one big knowledge base, I split project knowledge into three separate RAG sources. That one decision did more for output quality than any amount of extra context ever did. I stopped thinking of RAG as "extra documents" at some point and started thinking of each source as a different kind of organizational memory. Release notes are the record of what's already shipped: completed features, closed bugs, delivered capabilities. Feed them to the analyzer and it stops recommending work that's already done. It can also reason about how the project evolved release over release, instead of judging every ticket on its own. This one is trickier to pin down because it's the stuff experienced team members carry around in their heads and almost never write into a ticket. Program goals, system architecture, engineering constraints, stakeholder priorities, the domain jargon nobody bothers to define. Without it, the model can tell you what a ticket says but not why anyone should care. Once I added program context, cluster descriptions got noticeably sharper, because the model could finally connect a ticket to the bigger system it belonged to. The last RAG is roadmap information and the strategic themes for the current planning cycle. Release notes look backward; this one looks forward. It ended up mattering most for executive summaries. Instead of just describing what was sitting in the backlog, the analyzer could weigh whether a cluster of tickets actually lined up with where the program was trying to go. Recommendations started reflecting the destination, not just the history. Each source lives in its own vector index. Release notes, program docs, and roadmap docs are chunked and embedded separately, then pulled through a LangChain retrieval chain scoped to whatever task is running. Keeping them apart instead of merging into one index turns out to matter a lot. Similarity search over a combined index tends to surface whichever source happens to have the most chunks, which usually buries the roadmap themes under a mountain of release-note text. Three smaller indexes, each scoped to a task, just retrieve more precisely than one big one ever will. Here's the lesson that took me a bit too long to learn: more context isn't automatically better context. At first I fed all three RAG sources into every single prompt. Seemed thorough. In practice it just diluted the useful context with noise. So I matched sources to task instead: Cutting prompts down this way improved output quality while making them smaller. Sometimes the better engineering move isn't adding more information. It's adding the right information and leaving the rest out. | Task | Primary RAG | Background | |---|---|---| | Cluster naming & PM recommendations | 2026 theme.md | project context.md | | Duplicate insights | release notes.md | project context.md | | Executive summary | 2026 theme.md & release notes.md | project context.md | One thing surprised me during evaluation. I compared outputs across multiple LLMs, scored by both automated LLM judges and human reviewers. The judges didn't always agree with each other on quality, but they agreed on this: outputs grounded in real project knowledge beat generic software advice, every time. And the improvement wasn't that summaries got longer or more detailed. They got more specific. Instead of recommending "improve system performance," the analyzer could tie a recommendation to an actual roadmap priority, a release that had already shipped, or an engineering initiative already underway. That specificity is what project managers actually want, not more words. I don't think of what I built as "adding RAG" anymore. I think of it as organizing institutional knowledge: Splitting those responsibilities apart made prompt design simpler, cut out a lot of unnecessary context, and led to recommendations that actually matched what the project needed. This isn't specific to Jira backlogs, either. Any enterprise LLM application probably has several categories of institutional knowledge floating around, policies, operating procedures, historical records, future plans, and each one might deserve its own retrieval strategy rather than getting dumped into one shared index. As these systems mature, I suspect the real gains won't come from retrieving more documents. They'll come from organizing what you retrieve more thoughtfully. One RAG is a fine place to start. It just isn't always where you should end up. RAG files are on GitHub https://github.com/wkgann/Jira-Backlog-Analyzer-part-2/tree/main/Jira%20Backlog%20Analyzer-part2/rags .