Chat With Your Documents Locally Using AnythingLLM and Ollama

A developer built a private RAG system using AnythingLLM and Ollama that runs locally on any machine, allowing users to drop in PDFs, Word docs, and code files and ask questions without cloud dependency. The setup uses Qwen3 14B as the default model and nomic-embed-text for embeddings, with options for desktop or Docker deployment.

A private RAG system where you drop in PDFs, Word docs, and code files and ask questions. Runs on any machine, no cloud dependency. | Component | Role | |---|---| | AnythingLLM | Desktop/server app with RAG, agents, built-in vector DB | | Ollama | Serves local LLM for chat + embeddings | | Qwen3 14B | Default model for answering questions | Install from ollama.com, or run with Docker: docker run -d --gpus all -p 11434:11434 --name ollama \ -v ollama:/root/.ollama ollama/ollama Pull a model: ollama pull qwen3:14b Pull an embedder: ollama pull nomic-embed-text Desktop app easiest : Download from anythingllm.com Docker: docker run -d -p 3001:3001 --name anythingllm \ --add-host host.docker.internal:host-gateway \ -v anythingllm:/app/server/storage \ mintplexlabs/anythingllm | Local | ChatGPT + GPTs | | |---|---|---| | Monthly | $0 | $20-200 | | Hardware | $0-300 | $0 | | Privacy | Stays on your machine | Sent to cloud | | Documents | Unlimited | Token-limited | Full guide with troubleshooting: https://everylocalai.com/stack/anythingllm-ollama-rag https://everylocalai.com/stack/anythingllm-ollama-rag