I'm building a SaaS platform in Sri Lanka that handles documents and other sensitive data.
Each user can upload their own documents and information, and the platform uses RAG to answer questions based on that user's data. That part makes sense to me.
My main concern is what happens when the user hasn't uploaded enough information. I still want the LLM to provide accurate answers using reliable information from the internet (or from a curated knowledge base), with proper citations.
These are the two architectures I'm considering:
Option 1:
Base LLM (OpenAI/Anthropic via Azure AI Foundry or Amazon Bedrock) ↓ Platform RAG (global knowledge base managed by us) ↓ User-specific RAG In this approach, we maintain a global knowledge base that we (the platform admins) curate and update. Every user can access this shared knowledge, while their own uploaded documents are searched through their personal RAG.
Option 2:
Open-source LLM ↓ Fine-tuned on Sri Lankan/domain-specific data ↓ User-specific RAG Here, we fine-tune an open-source model using Sri Lankan or domain-specific data, and each user still has their own RAG for their private documents.
My concerns are:
Is fine-tuning actually the right solution here, or is it unnecessary?
Is a global/shared RAG a better approach than fine-tuning?
How would you design this architecture if you wanted:
Accurate answers from domain knowledge
User-private document search
Citations/sources
Good scalability for thousands of users
I'm leaning toward Option 1 because fine-tuning seems expensive, time-consuming, and I have no experience with it yet. However, I'm not sure if I'm thinking about this correctly.
I'd really appreciate hearing how others would approach this problem
Comments URL: [https://news.ycombinator.com/item?id=49059969](https://news.ycombinator.com/item?id=49059969)
Points: 1