Show HN: Raggy – A lightweight CLI tool for RAG over local documents

Developer paulknysh released Raggy, a lightweight CLI tool for retrieval-augmented generation (RAG) over local documents, built with LangChain, Chroma, and Ollama. Raggy runs a hybrid vector and BM25 index with local embedding generation, supports PDF, DOCX, PPTX, TXT, MD, Markdown, HTML, and image formats via OCR, and can generate answers either through a local LLM or remotely via OpenAI, Anthropic, or Google API keys. The tool requires Python 3.10 or newer and is installed by cloning the GitHub repository and running an editable install with uv or pipx.

A lightweight CLI tool for Retrieval-Augmented Generation RAG over local documents built with LangChain, Chroma, and Ollama. Hybrid database vector + BM25 index and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. raggy supports most common document formats and handles images/scans automatically via OCR. Usage example -- CLI returns an answer based on your documents, and citations along with their locations and relevance scores: These are all currently supported file formats all other formats are ignored : | Type | Extensions | |---|---| | Documents | .pdf , .docx , .pptx | | Text | .txt , .md , .markdown | | Web | .html , .htm | | Images OCR | .png , .jpg , .jpeg , .bmp | Ollama is required for running the local embedding model which feeds the on-disk vector DB , and also a local LLM if needed . To install Ollama: curl -fsSL https://ollama.com/install.sh | sh may need to start ollama after installation using the app or: ollama or ollama serve If an API key will be used for accessing an LLM remotely, a standard environment variable needs to be set one of the following : export GEMINI API KEY=... export OPENAI API KEY=... export ANTHROPIC API KEY=... Python 3.10 or newer is required; installing uv is recommended: curl -LsSf https://astral.sh/uv/install.sh | sh Clone the repo: git clone https://github.com/paulknysh/raggy.git && cd raggy Then install using: with uv uv tool install -e . with pipx pipx install -e . For now, cloning + editable install is picked as a preferred installation method, as it allows you to experiment with the demo dataset, run the eval harness, and edit/debug code if needed. In the future, direct install via uv tool install git+https ... / pipx install git+https ... will be used instead. First, run this command inside the cloned repo: make config It creates your own user config config.yaml where all your execution parameters live. While config.yaml comes with defaults you can test, you should populate sources your input folders/files and db directory DB location sections with your preferred paths. For a detailed overview of all config parameters, see Configuration configuration . To start the CLI, use the raggy <path-to-config-file command: raggy config.yaml Important CLI automatically pulls all models listed in config.yaml and re- indexes your documents -- this might take a while on the first run, depending on models chosen, document count/size, and whether OCR is needed scans, images, etc . Important Relative paths in config.yaml resolve against the current directory from where raggy command is executed . Keep that in mind if you want to run raggy from other locations. To be safe, just always use absolute paths in your config file. Here is the basic snippet you can run via uv run snippet.py : python from raggy import run pipeline, source label query = "What is TS-RAG?" response, retrieved docs = run pipeline query, config path="config.yaml" print f"\n RESPONSE:\n\n{response}\n\n " for i, doc in enumerate retrieved docs, 1 : print f"\n\n=== Doc {i} {source label doc } ===\n\n" print doc.page content All runtime settings are defined in the config file: | Setting | Description | |---|---| | sources | list of source directories and/or files | | db directory | location where the DB itself is stored | | embedding model | Ollama embedding model e.g. nomic-embed-text | | chunk size | chunk size in characters | | chunk overlap | character overlap between adjacent chunks | | embed batch size | max number of chunks embedded per batch into Chroma 100 in the shipped config ; the number of batches is derived automatically | | llm provider | where generation runs: ollama local, the shipped value or openai / anthropic / google via API | | llm model | chat model for generation e.g. phi4-mini locally, or a remote model name like gemini-3.7-flash | | llm temperature | LLM sampling temperature | | retrieve k | total chunks retrieved per query, split across the dense and lexical retrievals 50 in the shipped config | | hybrid alpha | fraction of retrieve k spent on the vector retrieval; the remainder goes to lexical 1.0 = vector only, 0.0 = lexical only, 0.5 in the shipped config | | rerank model | Hugging Face ID of the cross-encoder model e.g. cross-encoder/ms-marco-MiniLM-L6-v2 | | rerank k | number of chunks returned by the cross-encoder must be <= retrieve k | | rerank threshold | drops reranked chunks whose relevance score is below this value 0.0 = disabled, 0.3 in the shipped config | | system prompt | system prompt dictating how the LLM should answer; must contain a {context} placeholder | Notes: - The current default config parameters were tested on a basic MacBook Air with 8GB RAM. Switching to much heavier local models likely needs appropriate GPU/memory. - Chroma doesn't seem to be able to embed all chunks in one go; therefore, embed batch size was introduced so it's done in batches instead. 100 seems like a reasonable default, but if you get Chroma errors during embedding such as Error: Post "http://127.0.0.1:50175/tokenize": EOF status code: 400 , try lowering embed batch size further. Below are the main steps in the RAG pipeline assuming the DB is already created : 1 Hybrid retrieval. retrieve k is a total candidate budget , split by hybrid alpha between two retrievals over the whole corpus: - dense retrieval -- nearest chunks in Chroma by embedding similarity good at paraphrase and synonyms ; - lexical retrieval -- the persisted bm25s index good at exact terms: identifiers, names, acronyms, numbers . So retrieve k: 50 with hybrid alpha: 0.5 takes 25 chunks from each arm. The two ranked lists are merged by reciprocal rank fusion , which collapses duplicates and needs no score calibration between the two very different scales. Fusion weights are uniform on purpose: hybrid alpha already sets each arm's influence by deciding how many candidates it contributes. 2 Cross-encoder reranking. Stage 1 favors recall and is noisy. The reranker rerank model , run locally on onnxruntime pushes the query and the chunk through the model together and emits one relevance score per pair -- far sharper than cosine distance between separately embedded texts, and affordable on ~50 chunks though not on the whole corpus. The top rerank k chunks survive. 3 Score threshold. Each chunk carries its reranker score in doc.metadata "relevance score" , and anything below rerank threshold is dropped. This keeps rerank k from polluting the context when the corpus has no good answer; 0.0 disables it. 4 Generation. The survivors are concatenated into {context} in system prompt and sent to the configured LLM, along with the chat history in chat mode and the question. The exact chunks the model saw are returned to the caller -- retrieved docs above, and the source table the CLI prints. DB creates/updates itself automatically, so you don't need to think about it. Below is just a high-level overview of the mechanics. On the first run, initialize db in raggy/indexing.py loads every supported file under each entry in sources , splits them into overlapping chunks, and embeds them into Chroma. It also writes a manifest.yaml into the persist directory recording these parameters: - sources - chunk size - chunk overlap - embedding model - files — a {file path: SHA-256 content hash} map of every indexed file The same chunks are also indexed with bm25s a lexical BM25 index , stored in <db directory /bm25 index/ , so the lexical half of hybrid retrieval runs without re-indexing at retrieval time. The manifest.yaml is cheap to recompute, so for every new run it is computed and compared against the existing one. What happens next depends on what changed: - Incremental update the common case — the source files changed but chunk size , chunk overlap , and embedding model did not. In this case, only added and modified files are reloaded and embedded. Untouched files are never re-embedded, so editing one file in a large corpus costs one file's worth of work. - Full rebuild — chunk size , chunk overlap , or embedding model changed every stored vector is then invalid , or there is no manifest yet. The persist directory is wiped, and everything is indexed from scratch. The BM25 index has no incremental update path, so it is rebuilt after every update — from the chunks already stored in Chroma, which needs no embedding calls and no re-reading of source files. This should be very fast anyway. This article https://arxiv.org/abs/2608.06223v1 https://arxiv.org/abs/2608.06223v1 is used here as a demo dataset. It's an 8-page document -- each page is saved in different file formats including PDF, plaintext, images, MS Office and saved inside the sample docs directory. This directory is specified in config.yaml by default. eval folder currently contains a basic harness to test pipeline performance on the demo dataset. You can run it by: uv run eval/run eval.py It computes basic retrieval/generation metrics and produces a summary both printed and saved to eval/results.json . The Q&A pairs are about sample docs , so the harness always runs against default config/default config.yaml rather than your own config.yaml . If some features are not working or missing, feel free to open an issue or a PR. MIT