Demo stack: LangChain, Llama.cpp, Mistral.ai, Qwen Embeddings, PostgreSQL, Telegraf, Prometheus on Docker.
I recently came across Machine Learning Mastery's guide on Building a RAG Pipeline with llama.cpp and decided to try it myself.
One thing quickly became apparent: some of the APIs and methods used in the example had already changed or become deprecated.
That led me to a bigger question:
How do you design a RAG system that can evolve as the underlying technologies change?
Instead of tightly coupling the application to a specific LLM, vector database, or document-processing framework, I experimented with a more modular architecture.
This way I get a more resilient architecture, for instance if I need a transition between local on-premise to fully on cloud or just hybrid; ability to switch to different database like PostgreSQL instead of ChromaDB, due to the data integration and ACID compliance!
A few principles became particularly important:
def importOnCall(..):
from.. import .. # Lazy import
return ResultModel(..) # tied to Abstracts & Models
vectorstore: VectorStore = PGVectorStore(...)
vectorstore: VectorStore = OtherVectorDBService(..)
🛠️ Separation of responsibilities
Each layer has a focused responsibility:
📦️ Observability, Containerization & Resource/Network monitoring, Confiurations are pass in form of Environment Variables, see sample.env. Telegraf is used to monitor the resource consumsion and network traffic, and project metrics to Prometheus. See the 👁️ Observability section.
The complete implementation is available on GitHub: yoga1290/rag