Building an AI Customer Support SaaS with Django, RAG and Self-Hosted LLMs

An engineer has built AI-Autofy, an AI customer-support SaaS using Django, RAG, and self-hosted LLMs. The architecture separates the AI inference layer from the main web application to allow independent scaling, and uses a retrieval pipeline with tenant isolation to ground answers in business-specific data. The developer highlights the importance of relevance gating and distinguishing static knowledge from live data for accurate responses.

Building an AI Customer Support SaaS with Django, RAG and Self-Hosted LLMs Over the past several months, I’ve been building AI-Autofy, an AI customer-support SaaS designed to let businesses train an assistant on their own website, documents, FAQs and business data. At first glance, building an AI chatbot sounds straightforward: Send a prompt to an LLM. Display the response. Add a chat widget. In practice, once you need reliable business-specific answers, tenant isolation, live data, product information, images, analytics and predictable inference costs, the architecture becomes considerably more interesting. This post covers some of the main lessons I learned while building it. The basic architecture The web application is built with Python and Django. Django handles things such as: Customer accounts Subscriptions AI configuration Knowledge-base management Chat history Analytics Widget configuration Tenant separation Integrations The AI inference layer is separated from the main Django application. This means the web application does not need to run the language model itself. Instead, requests are sent to an AI service responsible for generating responses. Why separate the AI service? Running an LLM inside the main web application creates several problems. Inference workloads have very different requirements from normal web requests. A typical Django request might take milliseconds, while an AI response may involve: Retrieval Prompt construction GPU inference Streaming tokens Tool calls Live-data lookups Separating these workloads allows the web application and AI infrastructure to scale independently. It also makes it possible to change the model without redesigning the SaaS application. Retrieval-Augmented Generation A customer-support assistant should not rely entirely on the model’s general knowledge. A business wants the AI to answer questions using its own information. For example: What is your refund policy? or: Do you provide support outside Ireland? The relevant information might exist on the company website or inside a PDF. I therefore use a retrieval pipeline. Conceptually: Customer question | v Create embedding | v Search business knowledge | v Retrieve relevant documents | v Build LLM prompt | v Generate grounded answer The important part is tenant isolation. A document belonging to Company A must never appear in a response generated for Company B. Every retrieval request therefore needs to remain scoped to the current tenant. A vector database is only part of the solution I use a vector database for semantic retrieval, but retrieval quality depends on much more than simply storing embeddings. Things that matter include: Chunk size Metadata Tenant filtering Similarity thresholds Query rewriting Number of retrieved chunks Prompt construction Retrieving too little context can produce incomplete answers. Retrieving too much can fill the context window with irrelevant information. I found that relevance filtering is one of the most important parts of the system. Static knowledge versus live data A vector database works well for relatively static information. But consider a question such as: What products are currently available? That information may change constantly. Embedding yesterday’s product catalogue is not necessarily the right solution. I therefore treat knowledge data and live data differently. Knowledge data includes things such as: Website content FAQs Documentation Policies Live data can include: Products Prices Availability Business-system information The interesting problem is deciding when live data should be queried. You do not want a product catalogue added to every prompt simply because it exists. The user’s question needs to be relevant first. Relevance gating This turned out to be an important lesson. Imagine a customer asks: What are your opening hours? If the application automatically injects product data into every request, the LLM may start mentioning products even though the question has nothing to do with them. The same applies to images. The solution is to introduce relevance checks before enriching the prompt. Conceptually: if is product question message : context += get product data The real implementation can be more sophisticated, but the principle is simple: Give the model additional information only when it is relevant. This improves both response quality and token efficiency. Dynamic images have the same problem AI responses can become much more useful when they contain relevant images. For example, if somebody asks: Can you show me the blue version? an image may be very helpful. But displaying an image because a generic keyword happened to match makes the assistant feel unreliable. So image selection also needs relevance filtering. A useful AI interface is not about showing everything available. It is about showing the right information at the right moment. Self-hosting the language model One of the biggest architectural decisions was how to handle inference. Using hosted AI APIs is extremely convenient, particularly during development. However, predictable SaaS pricing becomes harder when every customer interaction has a variable external API cost. I therefore experimented with self-hosted models running on GPU infrastructure. The architecture is roughly: Website Widget | v Django | v AI Service / Agent | +---- Vector Database | +---- Live Data | +---- LLM Inference This gives more control over: Model choice Token limits Capacity Cost per message Scaling Data flow There are trade-offs, of course. Running inference infrastructure means dealing with GPU availability, model loading, monitoring and capacity planning. Streaming responses For chat applications, perceived latency matters almost as much as total generation time. Waiting several seconds and then receiving an entire response feels much slower than seeing the answer appear incrementally. Streaming therefore makes a significant difference to the user experience. The flow becomes: Browser | | question v Django | v AI service | | token stream v Django | | streamed response v Browser Even when total generation time remains similar, the application feels considerably more responsive. Human escalation still matters An AI support system should not pretend it can solve everything. There are situations where a human should take over. Examples include: Complaints Sensitive account issues Missing information Complex requests Situations requiring human judgement One of the design principles I have adopted is: The AI should know when it does not have enough information. That is more useful than confidently inventing an answer. Multitenancy changes everything Building an AI demo for one business is relatively easy. Building a SaaS where hundreds of businesses can independently configure their assistants is different. Each tenant may have: Different instructions Different knowledge Different products Different widgets Different usage limits Different conversation histories Every step of the pipeline must preserve tenant context. That includes retrieval, live-data access, logging and analytics. Cost becomes an architectural feature When building a SaaS product, AI cost is not just an infrastructure concern. It directly affects the business model. If a customer pays €20 per month, the platform cannot consume €30 of inference infrastructure serving that customer. This means decisions such as these become important: Context length Number of retrieved documents Model size GPU utilization Message limits Caching Prompt size Concurrency AI efficiency becomes part of product engineering. What I would do differently If I were starting again, I would spend more time on relevance and retrieval quality earlier. It is tempting to focus on model size. But for a customer-support system, a smaller model with excellent business context can often be more useful than a larger model receiving poor context. I would prioritize: Good retrieval Strong tenant isolation Relevance gating Clear system instructions Fast streaming Reliable fallbacks before spending too much time experimenting with larger models. The result These ideas eventually became part of AI-Autofy. The platform lets businesses add their website, documents, FAQs and instructions, configure an AI assistant, test it and deploy it using a website widget. I’ve also been adding capabilities for live business information, products, images, multilingual conversations, escalation and analytics. You can see the project here: I’m continuing to work on both the product and the infrastructure behind it. For anyone else building AI SaaS products, I’d be interested to hear how you are approaching the same trade-off between model quality, inference cost and retrieval quality.