A private RAG system where you drop in PDFs, Word docs, and code files and ask questions. Runs on any machine, no cloud dependency.
| Component | Role |
|---|---|
| AnythingLLM | Desktop/server app with RAG, agents, built-in vector DB |
| Ollama | Serves local LLM for chat + embeddings |
| Qwen3 14B | Default model for answering questions |
docker run -d --gpus all -p 11434:11434 --name ollama \
-v ollama:/root/.ollama ollama/ollama
ollama pull qwen3:14b
ollama pull nomic-embed-text
Desktop app (easiest): Download from anythingllm.com
Docker:
docker run -d -p 3001:3001 --name anythingllm \
--add-host host.docker.internal:host-gateway \
-v anythingllm:/app/server/storage \
mintplexlabs/anythingllm
| Local | ChatGPT + GPTs | |
|---|---|---|
| Monthly | $0 | $20-200 |
| Hardware | $0-300 | $0 |
| Privacy | Stays on your machine | Sent to cloud |
| Documents | Unlimited | Token-limited |
Full guide with troubleshooting: https://everylocalai.com/stack/anythingllm-ollama-rag
source & further reading
dev.to — original article
AI Pair Programming in Your Terminal with Aider and Ollama
Build a Unified AI Gateway with LiteLLM and Ollama
While My Friends Were Playing Games, I Was Pressing BREAK