{"slug": "building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms", "title": "Building an AI Customer Support SaaS with Django, RAG and Self-Hosted LLMs", "summary": "An engineer has built AI-Autofy, an AI customer-support SaaS using Django, RAG, and self-hosted LLMs. The architecture separates the AI inference layer from the main web application to allow independent scaling, and uses a retrieval pipeline with tenant isolation to ground answers in business-specific data. The developer highlights the importance of relevance gating and distinguishing static knowledge from live data for accurate responses.", "body_md": "Building an AI Customer Support SaaS with Django, RAG and Self-Hosted LLMs\n\nOver the past several months, I’ve been building AI-Autofy, an AI customer-support SaaS designed to let businesses train an assistant on their own website, documents, FAQs and business data.\n\nAt first glance, building an AI chatbot sounds straightforward:\n\nSend a prompt to an LLM.\n\nDisplay the response.\n\nAdd a chat widget.\n\nIn practice, once you need reliable business-specific answers, tenant isolation, live data, product information, images, analytics and predictable inference costs, the architecture becomes considerably more interesting.\n\nThis post covers some of the main lessons I learned while building it.\n\nThe basic architecture\n\nThe web application is built with Python and Django.\n\nDjango handles things such as:\n\nCustomer accounts\n\nSubscriptions\n\nAI configuration\n\nKnowledge-base management\n\nChat history\n\nAnalytics\n\nWidget configuration\n\nTenant separation\n\nIntegrations\n\nThe AI inference layer is separated from the main Django application.\n\nThis means the web application does not need to run the language model itself.\n\nInstead, requests are sent to an AI service responsible for generating responses.\n\nWhy separate the AI service?\n\nRunning an LLM inside the main web application creates several problems.\n\nInference workloads have very different requirements from normal web requests.\n\nA typical Django request might take milliseconds, while an AI response may involve:\n\nRetrieval\n\nPrompt construction\n\nGPU inference\n\nStreaming tokens\n\nTool calls\n\nLive-data lookups\n\nSeparating these workloads allows the web application and AI infrastructure to scale independently.\n\nIt also makes it possible to change the model without redesigning the SaaS application.\n\nRetrieval-Augmented Generation\n\nA customer-support assistant should not rely entirely on the model’s general knowledge.\n\nA business wants the AI to answer questions using its own information.\n\nFor example:\n\nWhat is your refund policy?\n\nor:\n\nDo you provide support outside Ireland?\n\nThe relevant information might exist on the company website or inside a PDF.\n\nI therefore use a retrieval pipeline.\n\nConceptually:\n\nCustomer question\n\n|\n\nv\n\nCreate embedding\n\n|\n\nv\n\nSearch business knowledge\n\n|\n\nv\n\nRetrieve relevant documents\n\n|\n\nv\n\nBuild LLM prompt\n\n|\n\nv\n\nGenerate grounded answer\n\nThe important part is tenant isolation.\n\nA document belonging to Company A must never appear in a response generated for Company B.\n\nEvery retrieval request therefore needs to remain scoped to the current tenant.\n\nA vector database is only part of the solution\n\nI use a vector database for semantic retrieval, but retrieval quality depends on much more than simply storing embeddings.\n\nThings that matter include:\n\nChunk size\n\nMetadata\n\nTenant filtering\n\nSimilarity thresholds\n\nQuery rewriting\n\nNumber of retrieved chunks\n\nPrompt construction\n\nRetrieving too little context can produce incomplete answers.\n\nRetrieving too much can fill the context window with irrelevant information.\n\nI found that relevance filtering is one of the most important parts of the system.\n\nStatic knowledge versus live data\n\nA vector database works well for relatively static information.\n\nBut consider a question such as:\n\nWhat products are currently available?\n\nThat information may change constantly.\n\nEmbedding yesterday’s product catalogue is not necessarily the right solution.\n\nI therefore treat knowledge data and live data differently.\n\nKnowledge data includes things such as:\n\nWebsite content\n\nFAQs\n\nDocumentation\n\nPolicies\n\nLive data can include:\n\nProducts\n\nPrices\n\nAvailability\n\nBusiness-system information\n\nThe interesting problem is deciding when live data should be queried.\n\nYou do not want a product catalogue added to every prompt simply because it exists.\n\nThe user’s question needs to be relevant first.\n\nRelevance gating\n\nThis turned out to be an important lesson.\n\nImagine a customer asks:\n\nWhat are your opening hours?\n\nIf the application automatically injects product data into every request, the LLM may start mentioning products even though the question has nothing to do with them.\n\nThe same applies to images.\n\nThe solution is to introduce relevance checks before enriching the prompt.\n\nConceptually:\n\nif is_product_question(message):\n\ncontext += get_product_data()\n\nThe real implementation can be more sophisticated, but the principle is simple:\n\nGive the model additional information only when it is relevant.\n\nThis improves both response quality and token efficiency.\n\nDynamic images have the same problem\n\nAI responses can become much more useful when they contain relevant images.\n\nFor example, if somebody asks:\n\nCan you show me the blue version?\n\nan image may be very helpful.\n\nBut displaying an image because a generic keyword happened to match makes the assistant feel unreliable.\n\nSo image selection also needs relevance filtering.\n\nA useful AI interface is not about showing everything available.\n\nIt is about showing the right information at the right moment.\n\nSelf-hosting the language model\n\nOne of the biggest architectural decisions was how to handle inference.\n\nUsing hosted AI APIs is extremely convenient, particularly during development.\n\nHowever, predictable SaaS pricing becomes harder when every customer interaction has a variable external API cost.\n\nI therefore experimented with self-hosted models running on GPU infrastructure.\n\nThe architecture is roughly:\n\nWebsite Widget\n\n|\n\nv\n\nDjango\n\n|\n\nv\n\nAI Service / Agent\n\n|\n\n+---- Vector Database\n\n|\n\n+---- Live Data\n\n|\n\n+---- LLM Inference\n\nThis gives more control over:\n\nModel choice\n\nToken limits\n\nCapacity\n\nCost per message\n\nScaling\n\nData flow\n\nThere are trade-offs, of course.\n\nRunning inference infrastructure means dealing with GPU availability, model loading, monitoring and capacity planning.\n\nStreaming responses\n\nFor chat applications, perceived latency matters almost as much as total generation time.\n\nWaiting several seconds and then receiving an entire response feels much slower than seeing the answer appear incrementally.\n\nStreaming therefore makes a significant difference to the user experience.\n\nThe flow becomes:\n\nBrowser\n\n|\n\n| question\n\nv\n\nDjango\n\n|\n\nv\n\nAI service\n\n|\n\n| token stream\n\nv\n\nDjango\n\n|\n\n| streamed response\n\nv\n\nBrowser\n\nEven when total generation time remains similar, the application feels considerably more responsive.\n\nHuman escalation still matters\n\nAn AI support system should not pretend it can solve everything.\n\nThere are situations where a human should take over.\n\nExamples include:\n\nComplaints\n\nSensitive account issues\n\nMissing information\n\nComplex requests\n\nSituations requiring human judgement\n\nOne of the design principles I have adopted is:\n\nThe AI should know when it does not have enough information.\n\nThat is more useful than confidently inventing an answer.\n\nMultitenancy changes everything\n\nBuilding an AI demo for one business is relatively easy.\n\nBuilding a SaaS where hundreds of businesses can independently configure their assistants is different.\n\nEach tenant may have:\n\nDifferent instructions\n\nDifferent knowledge\n\nDifferent products\n\nDifferent widgets\n\nDifferent usage limits\n\nDifferent conversation histories\n\nEvery step of the pipeline must preserve tenant context.\n\nThat includes retrieval, live-data access, logging and analytics.\n\nCost becomes an architectural feature\n\nWhen building a SaaS product, AI cost is not just an infrastructure concern.\n\nIt directly affects the business model.\n\nIf a customer pays €20 per month, the platform cannot consume €30 of inference infrastructure serving that customer.\n\nThis means decisions such as these become important:\n\nContext length\n\nNumber of retrieved documents\n\nModel size\n\nGPU utilization\n\nMessage limits\n\nCaching\n\nPrompt size\n\nConcurrency\n\nAI efficiency becomes part of product engineering.\n\nWhat I would do differently\n\nIf I were starting again, I would spend more time on relevance and retrieval quality earlier.\n\nIt is tempting to focus on model size.\n\nBut for a customer-support system, a smaller model with excellent business context can often be more useful than a larger model receiving poor context.\n\nI would prioritize:\n\nGood retrieval\n\nStrong tenant isolation\n\nRelevance gating\n\nClear system instructions\n\nFast streaming\n\nReliable fallbacks\n\nbefore spending too much time experimenting with larger models.\n\nThe result\n\nThese ideas eventually became part of AI-Autofy.\n\nThe platform lets businesses add their website, documents, FAQs and instructions, configure an AI assistant, test it and deploy it using a website widget.\n\nI’ve also been adding capabilities for live business information, products, images, multilingual conversations, escalation and analytics.\n\nYou can see the project here:\n\nI’m continuing to work on both the product and the infrastructure behind it.\n\nFor anyone else building AI SaaS products, I’d be interested to hear how you are approaching the same trade-off between model quality, inference cost and retrieval quality.", "url": "https://wpnews.pro/news/building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms", "canonical_source": "https://dev.to/macliamor_d698380ee2bd235/building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms-3aoh", "published_at": "2026-08-15 16:22:55+00:00", "updated_at": "2026-08-15 16:42:36.355542+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-products", "developer-tools"], "entities": ["AI-Autofy", "Django", "RAG"], "alternates": {"html": "https://wpnews.pro/news/building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms", "markdown": "https://wpnews.pro/news/building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms.md", "text": "https://wpnews.pro/news/building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms.txt", "jsonld": "https://wpnews.pro/news/building-an-ai-customer-support-saas-with-django-rag-and-self-hosted-llms.jsonld"}}