# Taking RAG To Production: End-To-End Pitfalls From Actual Production Deployments > Source: > Published: 2026-09-15 03:59:28+00:00 I am going to admit something that is bad for my business. I run a no-code RAG platform. Building a working RAG chatbot on it really does take about ten minutes. Point it at a website, drop in some PDFs, and you have a ChatGPT-style assistant answering questions about your content. No vector database to babysit. No chunking strategy to agonize over. No embeddings pipeline to wire up at 2am. That part is real. It is genuinely a no-code afternoon. And it is also a lie. Not the platform. The ***demo***. Because the demo only ever shows you the **happy path**. The happy path is the 80 percent of cases that work the moment you press a button. The happy path is what makes everyone “happy”. Then the customer or manager says “yes”, and you meet the other 20 percent. This is a field report from that **other 20 percent**. We recently took a RAG deployment to production for a large national professional association, the kind that sits on decades of proprietary content their members pay real money to access. The happy path got us in the door. What followed was ten weeks of work across six fronts, almost none of which had anything to do with “the AI.” If you are about to take your own RAG project to production, build or buy, this is the map of where the time actually goes. Read it before you quote a timeline, to either your client OR your boss. Here is the thing nobody tells you in the demo. RAG is not a feature. RAG is a pipeline. And a pipeline is only as strong as its weakest stage. The demo shows you the middle of the pipeline, where the magic lives: retrieve some chunks, stuff them in a prompt, generate a beautiful grounded answer with citations. That middle is solved. That middle is a commodity now (with ChatGPT GPTs or Claude Projects) Production is everything on either side of that middle. It is how the data gets *in* (cleanly, completely, legally). It is how the answer gets *out* (securely, to the right person, inside the right paywall). It is who *owns* the thing after launch when the customer emails you on a Tuesday because something looks off. I am going to walk you through six gates. To reach production, your deployment has to clear all six. Skip one and you do not have a product. You have a demo that happens to be in production, which is a very different and much worse thing. Here are the six, in order: ingestion, inference, security, analytics, procurement, maintenance. Only one of them is “the AI.” Let’s go through the gates. Everyone budgets for the model. Nobody budgets for the data. This is backwards. The model is the easy part. Getting clean, complete, authorized data into the system is where your first weeks go. Here are the problems we actually ran into. **Private data.** The valuable content was behind a login. Of course it was. That is the entire reason an association has a business: members pay for access. So step one was not “ingest the website.” Step one was building authenticated access to content that is, by design, not publicly reachable. Public data is the demo. Private data is the deal. They are not the same engineering problem. **There was no sitemap.** You cannot crawl what you cannot enumerate. The site had no usable sitemap, so before a single document got ingested, we had to generate one. This is the unglamorous plumbing that never shows up in a keynote, and it is mandatory. **The bot wall.** Their learning-management system actively blocked our crawler. Not maliciously. It was sitting behind a standard bot-protection layer that does not know the difference between a scraper stealing content and an authorized integration the customer explicitly asked for. We had to build a custom path to get past it, with the customer’s blessing, to reach content the customer *owned*. Modern web infrastructure is built to keep robots out, and allowlisting an authorized one is real coordination work, not a checkbox. You are now a robot the customer invited in, and the infrastructure still has to be taught to honor the invitation. **Garbage in.** Then the content itself fought back. Some of it was pristine. Some of it was a decade of blog cruft, duplicated boilerplate, and dead pages. And here is the rule that has held on every deployment I have run: if you ingest garbage, you retrieve garbage, and the customer blames the AI. Context cleanliness is not a nice-to-have. It is the difference between an assistant that sounds authoritative and one that confidently serves up junk. Somebody has to decide what *not* to ingest. That somebody is doing real work. **The videos had no transcripts.** A huge chunk of the value was locked inside video. Training sessions, recorded talks, member education. RAG cannot retrieve what it cannot read, and you cannot embed a video. The videos had no transcripts. So we built a custom crawler that pulls the audio from the video and runs it through speech-to-text transcription before anything else touches it. Your RAG system is only as good as the text it can see, and a lot of the world’s best content is currently trapped in audio. **Scale broke things.** Large media files did not flow politely through the standard upload path. When you are dealing with hundreds of megabytes to a gigabyte of video, “just upload it” stops being a sentence that means anything. We worked around it by transcribing first and ingesting the transcript, which is a fraction of the size and, conveniently, the only part RAG actually needs. The lesson: at production scale, the naive ingestion path falls over, and you need a different path that respects what the system actually consumes. **Metadata and the citation trick.** The last one is not plumbing. It is strategy disguised as plumbing. The customer wanted to ingest a document so the assistant could answer questions about it. But they did *not* want the citation to link to the document. They wanted the citation to link to the paywalled “buy it now” page for that document. That sounds like a one-line config change. It is not. The ingested chunk and its displayed source have to be decoupled by design. The metadata has to be overridable. The thing you ingest and the thing you cite are not the same URL, on purpose, so the citation can point a member to the purchase page rather than the raw document. Simple to describe, real engineering to support. That is seven distinct problems. All of them are “data ingestion.” None of them are “the AI.” And we have not generated a single answer yet. The data ingestion gauntlet: seven steps, each with its workaround Now we generate answers. This is the part the demo nailed. This is also where the data sins of Gate 1 come back with interest. **Garbage in, garbage out is not a cliche. It is a law.** Most of the retrieval-quality problems we hit at inference traced back to something we let into the index. Want better answers? Far more often than you would guess, the fix is upstream, in what you ingested, not in the prompt. Teams burn weeks tuning the generation step when the real bug is three stages back in the pipeline. Fix the data, and the “model problem” usually resolves on its own. **The events problem.** Here is a genuinely interesting failure. The association ran events, and the event information was scattered. No single page held a whole event. The date lived one place, the description another, the registration link somewhere else. When the system retrieved it, the pieces never assembled into a coherent whole, and the relevant context would not fit cleanly into the window. Members would ask a simple question about an upcoming event and get a fragmented non-answer. Notice what this is and what it is not. This is not the model inventing facts out of nothing. It is the model working from context that existed but was structurally impossible to retrieve in one piece. Missing context is the root cause here, not a lying model, and that distinction matters. I will come back to it. **We built a custom MCP for it.** The fix was not a better prompt. The fix was a custom Model Context Protocol integration that pulls structured event data directly from the customer’s event feed, on demand, at query time. Instead of hoping the right scattered chunks land in the context window, the assistant calls a tool that returns the whole event, clean and complete. When your content has a shape that retrieval cannot handle, you stop fighting retrieval and you give the model a tool instead. **The PDF viewer with a paywall.** The customer wanted citations to open a real PDF viewer inside the experience, and they wanted that viewer to show a few pages and then blur the rest behind the paywall. The member reads the answer, sees the source, sees a preview of it, and hits the paywall for the rest. Building a viewer that renders some pages and gates the others by entitlement is enterprise-grade plumbing — one that drives actual business outcomes like upsells. **Persona tuning.** Out of the box, the assistant was helpful in a generic way. Generic is not good enough when members are paying for domain expertise. We spent real cycles tuning the persona and the instructions so answers came back in the right voice, at the right altitude, citing the right things. This is craft. It does not automate. **Hallucination management.** Yes, we ran evals and testing for made-up answers, because you always should. But here is the honest part, and the part most vendors will not tell you: on this deployment, hallucination was barely the problem. The failures were almost all *missing* context, not invented facts. A RAG system that is wired to say “[I do not know](https://arxiv.org/abs/2608.26385)” when it lacks grounding is worth more than one that sounds confident. The scary failure mode is not the model lying. It is the model answering a question it should have refused, because the data to answer it correctly never made it through Gate 1. The assistant works. Now you have to put it somewhere real, in front of real members, without leaking anything. This is where “we have a working chatbot” meets “we are an enterprise with an identity provider and a legal team”. **SSO setup.** The members log in through the customer’s single sign-on. Standing that up is not a toggle. It is a coordinated effort across two engineering teams, the customer’s and yours, mapping identity correctly so the right people get in and **nobody else does**. It ate a meaningful slice of the timeline, and it was on the *critical* path, because without it there is no secure way to expose the assistant to members at all. **Rate limiting.** The moment you expose an LLM-backed endpoint to a member base, you have to think about abuse, cost, and runaway usage. Rate limiting is not optional infrastructure. It is the seatbelt you install before you let anyone drive. **Embedding inside a paywall, with JWT.** The assistant had to live *inside* the members-only portal, behind the paywall, embedded in the customer’s own pages. That means signed tokens, JWT handshakes, and a security configuration that guarantees only authenticated members reach the assistant. The embed is trivial. The embed ***that only paying members can reach*** is a security project. **RBAC through their identity provider.** Here is the elegant part. The customer needed role-based access control. Different member tiers should reach different assistants and different content. But the members do not exist as users in *our* system, and the customer did not want to create them there. So we built it so the customer’s own identity provider sends a custom attribute in the SSO response, and that attribute maps to a role on our side. The customer controls access entirely from their IDP. We never store the user. The deployment security layer: member to SSO to JWT embed to RBAC attribute to assistant **No seats, full control.** This has a useful consequence. Because we never provision the members as users in our system, the customer was able to cover **tens of thousands of members** without per-seat licensing (this is why Claude can never do this!), while still getting full role-based access control. To be clear, that is a pricing decision the architecture *enabled*, not one it forced. But it is a real example of how the identity model and the commercial model end up entangled. Get the identity model right and you give yourself options on both security and cost. **API-based inference.** And for the surfaces where an embed did not fit, inference was available directly through the API, so the customer could wire the assistant into their own applications on their own terms. None of this is “AI work.” All of it is mandatory. You do not ship to an enterprise without clearing Gate 3, and the people who clear it are not prompt engineers. They are identity, security, and infrastructure people. You launched. You are not done. You are barely started. Because right now you have tens of thousands of members about to ask questions you have never seen, and you have no idea which ones the assistant is quietly getting wrong. **Make the logged-in user legible.** Out of the box, a member arriving through SSO looked like just another anonymous guest in the analytics. That is useless. The customer wanted to know who was asking what, at the level of a real logged-in member, not a faceless guest. So we pushed the SSO identity into the analytics layer, so an authenticated member shows up as an authenticated member. You cannot improve a black box. Step one of improvement is making the right thing visible. **The launch is the start of the real backlog.** And this is the payoff. Once real members were really using it, the analytics showed us what they actually asked, where the assistant fell short, and which content was missing. You find out that members keep asking about a topic the assistant has no document for, or that a whole category of questions returns weak answers because the source material was never ingested. That is not a failure report. That is a specification, written for you, by the people paying for the product. The version that launched becomes the version that wins, and the only reason you can build the better one is that you instrumented the first one properly. Production is not the finish line. It is the moment your real backlog finally gets written. Here is the gate that blindsides technical founders, because it has nothing to do with technology and it can sink you anyway. **The pre-sales calls.** This did not close on one call. It took several, working through requirements, security questions, and stakeholder buy-in. Enterprise trust is not granted. It is accumulated, one call at a time. **A custom legal agreement.** Standard terms were not enough. The deal needed a custom legal agreement, including an opt-out clause that gave the customer a defined exit. A risk-averse buyer signs faster when they can see the door. The opt-out is not a weakness in the contract. It is the thing that lets a cautious institution say yes at all. **SOC-2, validated by their auditor.** The customer’s auditor wanted to talk. Not read a PDF of your compliance badge. *Talk.* A real human on a real call, validating that your security posture is what you say it is. If you cannot survive that conversation, the deal dies in legal no matter how good your retrieval is. And one quiet, important nuance here: not every piece of a custom deployment automatically lives inside your formal compliance boundary. The custom crawler, the bespoke integrations, the hosting choices, those have to be reasoned about explicitly, not waved through with an assumption. Knowing exactly what is and is not inside your compliance scope is itself part of clearing this gate honestly. **Weekly syncs.** Through all of it, weekly synchronization meetings with an account manager. White-glove is not a slogan. It is a calendar invite that recurs. I want to be blunt about this gate, because it is the one I see brilliant engineers ignore until it kills their timeline. You can have the best RAG system on earth and still not ship, because procurement is a gate exactly like ingestion is a gate. Budget for it. Staff for it. Respect it. The six gates to production: ingestion, inference, security, analytics, procurement, maintenance The last gate never closes. That is the whole point of it. A production RAG deployment is not a thing you ship and walk away from. The customer’s content changes. Their members ask new questions. New formats show up. The events feed shifts. Something that worked in May behaves differently in July. New LLM models get released, ***at an insane pace***. Somebody has to own all of that. This is the forward-deployed-engineer reality. The most successful enterprise deployments have a named human who understands the customer’s data, their goals, and their quirks, and who keeps the thing healthy over time. The custom solutions built in Gates 1 through 5 do not maintain themselves. And here is the most useful thing I learned on this deployment. The single biggest predictor of how much custom work an enterprise RAG deployment needs is not the technology. It is who the customer has internally and how much time they have. A customer with someone who deeply understands their own data and a bit of AI fluency needs almost no white glove. A customer without that person needs a lot. The platform does not change. The *people* on the other side change everything. Let me bring it home, because this is not a doom post. I am genuinely bullish on RAG. I run a platform that makes the easy part easy, and making the easy part easy is real, durable value. The ten-minute demo is not a trick. It is a head start. But the head start is not the race. Here is what I want you to walk away with. **The last 20 percent is the product.** You can read a Medium article and put together a quick RAG demo — that is easy. The other 20 percent, the dirty data and the bot walls and the SSO and the auditor calls and the forward-deployed engineer, is the part that is painful. That is not overhead on top of the product. That *is* the product. **Build or buy, someone owns the last mile.** If you build your own RAG stack, you own all six gates forever, including the ones that have nothing to do with AI and everything to do with crawlers, identity providers, and legal. If you buy, you are paying someone else to own them. The work does not vanish either way. The only question is whose calendar it lands on. **Respect the gates.** Every one of the six is a real gate, and skipping any of them does not get you to production faster. It gets you to a public failure faster. Ingestion is a gate. Inference is a gate. Security is a gate. Analytics is a gate. Procurement is a gate. Maintenance is a gate. Plan for all six or plan to be surprised by all six. So the next time you watch a flawless ten-minute RAG demo, including one of mine, enjoy it. Then ask the only question that matters for production. Who is going to own the other 20 percent? That answer is your real project plan. *I have spent the last 3.5 years building* *CustomGPT.ai**, a no-code RAG platform, and watching what happens when the demo meets the deployment. The deployment in this piece is anonymized from a large member-based professional association, but every pitfall is real and every workaround shipped. If you are taking RAG to production, I hope this saved you a few months of finding these gates the hard way.* [Taking RAG To Production: End-To-End Pitfalls From Actual Production Deployments](https://pub.towardsai.net/taking-rag-to-production-end-to-end-pitfalls-from-actual-production-deployments-42f0751c7a78) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.