# Chat With Your Documents Locally Using AnythingLLM and Ollama

> Source: <https://dev.to/everylocalai/chat-with-your-documents-locally-using-anythingllm-and-ollama-j6b>
> Published: 2026-06-14 21:53:08+00:00

A private RAG system where you drop in PDFs, Word docs, and code files and ask questions. Runs on any machine, no cloud dependency.

| Component | Role |
|---|---|
| AnythingLLM | Desktop/server app with RAG, agents, built-in vector DB |
| Ollama | Serves local LLM for chat + embeddings |
| Qwen3 14B | Default model for answering questions |

```
# Install from ollama.com, or run with Docker:
docker run -d --gpus all -p 11434:11434 --name ollama \
  -v ollama:/root/.ollama ollama/ollama

# Pull a model:
ollama pull qwen3:14b
# Pull an embedder:
ollama pull nomic-embed-text
```

**Desktop app (easiest):** Download from anythingllm.com

**Docker:**

```
docker run -d -p 3001:3001 --name anythingllm \
  --add-host host.docker.internal:host-gateway \
  -v anythingllm:/app/server/storage \
  mintplexlabs/anythingllm
```

| Local | ChatGPT + GPTs | |
|---|---|---|
| Monthly | $0 | $20-200 |
| Hardware | $0-300 | $0 |
| Privacy | Stays on your machine | Sent to cloud |
| Documents | Unlimited | Token-limited |

Full guide with troubleshooting: [https://everylocalai.com/stack/anythingllm-ollama-rag](https://everylocalai.com/stack/anythingllm-ollama-rag)