A bank in the Gulf gave me a constraint I did not expect. The conversation was going normally — documents, embeddings, vector search — until their CTO leaned forward and said, "None of this can touch a public cloud." Not the customer documents. Not the queries. Not even the embedding calls. Their compliance policy classified the loan-approval corpus as data that could not leave the building, and that was non-negotiable, so my entire mental default — "just call an API" — was dead on arrival.
The interesting part was what happened next. Their server room already had two GPU boxes sitting half-idle, running internal dashboards and a video-transcoding job. We built the entire RAG pipeline on those boxes. The project went live in seven weeks, and it taught me everything that is actually different about self-hosted RAG: the hardware math, the software choices, the failure modes, and the moment when self-hosting stops making sense.
In this article I am going to give you that entire blueprint — the same one I would hand to any team facing a data-residency constraint, a cost ceiling, or just a strong allergy to SaaS.
Self-hosted RAG means every stage of the pipeline runs on infrastructure you control, instead of API calls to a cloud provider. That includes three things that are easy to forget:
People routinely self-host the retrieval half and quietly keep calling a hosted LLM for generation. That is a valid configuration, and sometimes it is the right one. But it is not fully self-hosted, and it still ships your queries to a third party. If the requirement is "no data leaves the building," generation has to be local too. Know which of the two you are committing to before you start buying GPUs.
Here is the number that decides everything: your model size determines your GPU, and your GPU determines your budget. The good news is that the bar is lower than most people think.
bge-m3
or e5
run comfortably on CPU. A modern 8-core server can embed a 1,000-page corpus overnight and a few hundred chunks per second at query time. Do not buy a GPU for embeddings; you do not need one.The concrete stack that shipped at the bank: two used 3090s (24 GB each, bought for a fraction of a new card's price), 64 GB RAM, a 4 TB NVMe, and a 12-core CPU. Total hardware spend was under four thousand dollars. That ran a 13B model at acceptable latency for a team of about thirty users.
After seven weeks of real usage, here is the stack I would spec today:
| Layer | Pick | Why |
|---|---|---|
| Embedding model | ||
bge-m3 (or nomic-embed-text ) |
||
| Strong multilingual + retrieval results, runs on CPU | ||
| Vector store | pgvector (if you already run Postgres) or Qdrant | pgvector = one less service to operate; Qdrant = faster at scale |
| Generation server | vLLM (production) or Ollama (small team / single box) | vLLM is the standard for throughput; Ollama for simplicity |
| LLM | Llama 3.1 8B / Qwen 2.5 7B (or 13B if VRAM allows) | Open weights, quantizable, license-friendly |
| Orchestration | A thin Python service (FastAPI) | Avoid heavyweight frameworks unless you genuinely need them |
The one rule I will fight you on: do not build the pipeline around a framework's abstractions. FastAPI, a scheduler for ingestion, and a few functions for embed/retrieve/generate is a stack a junior can maintain and a senior can debug. Every RAG framework I have evaluated adds a second source of truth for your prompts and your chunking that you will eventually have to own anyway.
Documents land in a watched folder or an internal file service. An ingestion worker chunks them (I default to 700-token chunks with a 100-token overlap), embeds each chunk with the local model, and writes vectors to pgvector. The critical production detail: ingestion must be idempotent and versioned. The bank's compliance team edits documents constantly, and an unversioned index will quietly serve retired policy — the exact failure mode that motivated the whole project. Every ingest job records a source hash; re-ingest only when the hash changes.
import bge_m3_ # runs locally, CPU-friendly
def ingest(doc_id: str, chunks: list[str], conn):
embeddings = bge_m3_.embed(chunks)
with conn.cursor() as cur:
for chunk, vec in zip(chunks, embeddings):
cur.execute(
"INSERT INTO chunks (doc_id, content, embedding, ingested_at) "
"VALUES (%s, %s, %s, now())",
(doc_id, chunk, vec),
)
conn.commit()
Queries hit a FastAPI endpoint. It embeds the query, runs a pgvector similarity search (ORDER BY embedding <=> $1 LIMIT k
), and applies a metadata filter — in the bank's case, only documents the requester's role is allowed to see. Filtering before ranking is how self-hosted RAG answers the compliance question that sent the whole project to self-hosting in the first place.
A re-ranking step (a cross-encoder on the top-20 candidates, picking the top-5) costs ~50 ms on CPU and reliably fixes the "right answer retrieved but ranked second" problem. It is the cheapest quality win in the entire pipeline.
The generation call goes to the local vLLM endpoint, with the retrieved chunks in the prompt and the same grounding instruction I use everywhere: answer only from context, decline when the context is insufficient. The CTO's team wanted to see provenance, so every answer carries the document name and section of each chunk it used. With retrieval you get that for free — the chunks are right there.
Query ──▶ embed ──▶ pgvector top-20 ──▶ cross-encoder top-5 ──▶ vLLM ──▶ answer + citations
Let me give you the honest numbers, because every vendor blog post leaves these out:
The day we went live, the bank's questions worked beautifully and we had no idea how healthy the system was. That changed when I added five metrics to a dashboard, and they have been the operating dashboard ever since:
This is the piece that separates "we run RAG" from "we operate RAG." Self-hosting hands you the raw logs and the raw GPUs; if you do not watch the five numbers above, the first sign of trouble is a user complaint, and by then the answer has already been wrong for a week. Monitoring is not a luxury layer — it is the line item that makes self-hosting defensible to a board that asked why you did not just call an API.
Quantization quality surprises. I switched from 8-bit to 4-bit quantization to fit a model on one GPU and watched retrieval-then-answer quality dip on legal boilerplate. Fix: benchmark the exact document types before you commit to a quantization level, not after.
Vector drift after model upgrades. Upgrading the embedding model orphans every previously embedded vector — old and new vectors are not comparable, and search quality silently collapses. Fix: re-embed the corpus on every model change, and keep the embedding model pinned and versioned in your ingestion metadata.
One box, many jobs. The "idle GPU" that made the project possible is the same GPU the dashboards render on. When the video-transcoding job ran during business hours, answer latency tripled. Fix: schedule ingestion and batch jobs off-peak, or dedicate one GPU to inference.
The eval trap. We built a labeled test set of 200 real bank queries with the ground-truth chunk for each. Without it, every "improvement" to chunking or re-ranking was a guess. With it, we could prove the re-ranker was worth adding. Self-hosting does not excuse you from evaluation — it makes it easier to run, because you control the data.
Licensing and support. You are now the support line for a model you run yourself. Open-weights licenses vary (Llama has acceptable-use terms; others have different restrictions). Read them before you commit, and know that there is no vendor to call at 2 AM. Your team is the vendor now.
This is the honest part. Self-hosting RAG is not always the right answer, and the worst deployment I ever saw was someone who chose it for the wrong reason:
Self-host when: you have a hard data-residency or compliance requirement; your query volume and model needs are small enough that a local box beats the cloud bill; or you want full control over fine-tuning, versions, and provenance.
Do not self-host when: your only reason is "cloud is scary," your corpus is huge and your team has no one who can operate a GPU server, or your quality bar genuinely needs frontier models that do not fit on the hardware you are willing to buy. In that last case, the honest engineering answer is a hybrid — local retrieval for the private corpus, hosted frontier model for generation of the summarized answer — which violates zero compliance rules if the summarized output does not contain sensitive data. I have recommended exactly that to teams that asked me to "go fully self-hosted" for prestige. The best architecture is the one that meets the constraint at the lowest cost, and sometimes the constraint does not actually require self-hosting at all.
The bank's CTO got what he asked for: a RAG pipeline that never left the building, built on hardware that was already sitting there, answering compliance questions with citations the auditors could open. He also got something he did not ask for — a monthly maintenance line item, a quantified latency budget, and a team that now knows what "operating your own AI" actually costs.
Self-hosted RAG is not a hack and not a religion. It is a math problem: model size against VRAM, quality against latency, maintenance against vendor fees, and compliance against convenience. Run that math honestly, and the answer tells you exactly which stack to build.
*Gulshan Yad