{"slug": "show-hn-raggy-a-lightweight-cli-tool-for-rag-over-local-documents", "title": "Show HN: Raggy – A lightweight CLI tool for RAG over local documents", "summary": "Developer paulknysh released Raggy, a lightweight CLI tool for retrieval-augmented generation (RAG) over local documents, built with LangChain, Chroma, and Ollama. Raggy runs a hybrid vector and BM25 index with local embedding generation, supports PDF, DOCX, PPTX, TXT, MD, Markdown, HTML, and image formats via OCR, and can generate answers either through a local LLM or remotely via OpenAI, Anthropic, or Google API keys. The tool requires Python 3.10 or newer and is installed by cloning the GitHub repository and running an editable install with uv or pipx.", "body_md": "A lightweight CLI tool for Retrieval-Augmented Generation (RAG) over local documents built with LangChain, Chroma, and Ollama. Hybrid database (vector + BM25 index) and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. `raggy` supports most common document formats and handles images/scans automatically via OCR.\n\nUsage example -- CLI returns an answer based on your documents, and citations along with their locations and relevance scores:\n\nThese are all currently supported file formats (all other formats are ignored):\n\n| Type | Extensions | \n|---|---|\n| Documents | `.pdf` ,`.docx` ,`.pptx` | \n| Text | `.txt` ,`.md` ,`.markdown` | \n| Web | `.html` ,`.htm` | \n| Images (OCR) | `.png` ,`.jpg` ,`.jpeg` ,`.bmp` | \n\nOllama is required for running the local embedding model (which feeds the on-disk vector DB), and also a local LLM (if needed). To install Ollama:\n\n```\ncurl -fsSL https://ollama.com/install.sh | sh\n\n# may need to start ollama after installation using the app or:\nollama\n# or\nollama serve\n```\n\nIf an API key will be used for accessing an LLM remotely, a standard environment variable needs to be set (one of the following):\n\n```\nexport GEMINI_API_KEY=...\nexport OPENAI_API_KEY=...\nexport ANTHROPIC_API_KEY=...\n```\n\nPython 3.10 or newer is required; installing `uv` is recommended:\n\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\nClone the repo:\n\n```\ngit clone https://github.com/paulknysh/raggy.git && cd raggy\n```\n\nThen install using:\n\n```\n# with uv\nuv tool install -e .\n\n# with pipx\npipx install -e .\n```\n\nFor now, cloning + editable install is picked as a preferred installation method, as it allows you to experiment with the demo dataset, run the eval harness, and edit/debug code if needed. In the future, direct install via `uv tool install git+https ...`/` pipx install git+https ...` will be used instead.\n\nFirst, run this command inside the cloned repo:\n\n```\nmake config\n```\n\nIt creates your own user config (`config.yaml`) where all your execution parameters live. While `config.yaml` comes with defaults you can test, you should populate `sources` (your input folders/files) and `db_directory` (DB location) sections with your preferred paths. For a detailed overview of all config parameters, see [Configuration](#configuration).\n\nTo start the CLI, use the `raggy <path-to-config-file>` command:\n\n```\nraggy config.yaml\n```\n\nImportant\n\nCLI automatically pulls all models listed in `config.yaml` and (re-)indexes your documents -- this might take a while on the first run, depending on models chosen, document count/size, and whether OCR is needed (scans, images, etc).\n\nImportant\n\nRelative paths in `config.yaml` resolve against the current directory (from where `raggy` command is executed). Keep that in mind if you want to run `raggy` from other locations. To be safe, just always use absolute paths in your config file.\n\nHere is the basic snippet you can run via `uv run snippet.py`:\n\n``` python\nfrom raggy import run_pipeline, source_label\n\nquery = \"What is TS-RAG?\"\n\nresponse, retrieved_docs = run_pipeline(query, config_path=\"config.yaml\")\n\nprint(f\"\\n*** RESPONSE:\\n\\n{response}\\n\\n***\")\n\nfor i, doc in enumerate(retrieved_docs, 1):\n    print(f\"\\n\\n=== Doc {i} [{source_label(doc)}] ===\\n\\n\")\n    print(doc.page_content)\n```\n\nAll runtime settings are defined in the config file:\n\n| Setting | Description | \n|---|---|\n| `sources` | list of source directories and/or files | \n| `db_directory` | location where the DB itself is stored | \n| `embedding_model` | Ollama embedding model (e.g. `nomic-embed-text` ) | \n| `chunk_size` | chunk size in characters | \n| `chunk_overlap` | character overlap between adjacent chunks | \n| `embed_batch_size` | max number of chunks embedded per batch into Chroma ( `100` in the shipped config); the number of batches is derived automatically | \n| `llm_provider` | where generation runs: `ollama` (local, the shipped value) or`openai` /`anthropic` /`google` (via API) | \n| `llm_model` | chat model for generation (e.g. `phi4-mini` locally, or a remote model name like`gemini-3.7-flash` ) | \n| `llm_temperature` | LLM sampling temperature | \n| `retrieve_k` | total chunks retrieved per query, split across the dense and lexical retrievals ( `50` in the shipped config) | \n| `hybrid_alpha` | fraction of `retrieve_k` spent on the vector retrieval; the remainder goes to lexical (`1.0` = vector only,`0.0` = lexical only,`0.5` in the shipped config) | \n| `rerank_model` | Hugging Face ID of the cross-encoder model (e.g. `cross-encoder/ms-marco-MiniLM-L6-v2` ) | \n| `rerank_k` | number of chunks returned by the cross-encoder (must be `<= retrieve_k` ) | \n| `rerank_threshold` | drops reranked chunks whose relevance score is below this value ( `0.0` = disabled,`0.3` in the shipped config) | \n| `system_prompt` | system prompt dictating how the LLM should answer; must contain a `{context}` placeholder | \n\nNotes:\n\n- \nThe current default config parameters were tested on a basic MacBook Air with 8GB RAM. Switching to much heavier local models likely needs appropriate GPU/memory.\n- \nChroma doesn't seem to be able to embed all chunks in one go; therefore, `embed_batch_size` was introduced so it's done in batches instead. 100 seems like a reasonable default, but if you get Chroma errors during embedding (such as`Error: Post \"http://127.0.0.1:50175/tokenize\": EOF (status code: 400)` ), try lowering`embed_batch_size` further.\n\nBelow are the main steps in the RAG pipeline (assuming the DB is already created):\n\n**[1] Hybrid retrieval.** `retrieve_k` is a total *candidate budget*, split by\n`hybrid_alpha` between two retrievals over the whole corpus:\n\n- **dense** retrieval -- nearest chunks in Chroma by embedding similarity (good at\nparaphrase and synonyms);\n- **lexical** retrieval -- the persisted`bm25s` index (good at exact terms:\nidentifiers, names, acronyms, numbers).\n\nSo `retrieve_k: 50` with `hybrid_alpha: 0.5` takes 25 chunks from each arm. The two\nranked lists are merged by **reciprocal rank fusion**, which collapses duplicates and\nneeds no score calibration between the two very different scales. Fusion weights are\nuniform on purpose: `hybrid_alpha` already sets each arm's influence by deciding how\nmany candidates it contributes.\n\n**[2] Cross-encoder reranking.** Stage 1 favors recall and is noisy. The reranker\n(`rerank_model`, run locally on onnxruntime) pushes the query and the chunk through\nthe model *together* and emits one relevance score per pair -- far sharper than\ncosine distance between separately embedded texts, and affordable on ~50 chunks\nthough not on the whole corpus. The top `rerank_k` chunks survive.\n\n**[3] Score threshold.** Each chunk carries its reranker score in\n`doc.metadata[\"relevance_score\"]`, and anything below `rerank_threshold` is dropped.\nThis keeps `rerank_k` from polluting the context when the corpus has no good answer;\n`0.0` disables it.\n\n**[4] Generation.** The survivors are concatenated into `{context}` in\n`system_prompt` and sent to the configured LLM, along with the chat history (in chat\nmode) and the question. The exact chunks the model saw are returned to the caller --\n`retrieved_docs` above, and the source table the CLI prints.\n\nDB creates/updates itself automatically, so you don't need to think about it. Below is just a high-level overview of the mechanics.\n\nOn the first run, `initialize_db` (in `raggy/indexing.py`) loads every\nsupported file under each entry in `sources`, splits them into overlapping chunks,\nand embeds them into Chroma. It also writes a `manifest.yaml` into the persist\ndirectory recording these parameters:\n\n- `sources`\n- `chunk_size`\n- `chunk_overlap`\n- `embedding_model`\n- `files` — a`{file path: SHA-256 content hash}` map of every indexed file\n\nThe same chunks are also indexed with `bm25s` (a lexical BM25 index), stored in\n`<db_directory>/bm25_index/`, so the lexical half of hybrid retrieval runs\nwithout re-indexing at retrieval time.\n\nThe `manifest.yaml` is cheap to recompute, so for every new run it is computed and\ncompared against the existing one. What happens next depends on what changed:\n\n- **Incremental update (the common case)** — the source files changed but`chunk_size` ,`chunk_overlap` , and`embedding_model` did not. In this case, only added\nand modified files are reloaded and embedded. Untouched files are never\nre-embedded, so editing one file in a large corpus costs one file's worth of work.\n- **Full rebuild** —`chunk_size` ,`chunk_overlap` , or`embedding_model` changed\n(every stored vector is then invalid), or there is no manifest yet. The persist\ndirectory is wiped, and everything is indexed from scratch.\n\nThe BM25 index has no incremental update path, so it is rebuilt after every update — from the chunks already stored in Chroma, which needs no embedding calls and no re-reading of source files. This should be very fast anyway.\n\nThis article ([https://arxiv.org/abs/2608.06223v1](https://arxiv.org/abs/2608.06223v1)) is used here as a demo dataset. It's an 8-page document -- each page is saved in different file formats (including PDF, plaintext, images, MS Office) and saved inside the `sample_docs` directory. This directory is specified in `config.yaml` by default.\n\n`eval` folder currently contains a basic harness to test pipeline performance on the demo dataset. You can run it by:\n\n```\nuv run eval/run_eval.py\n```\n\nIt computes basic retrieval/generation metrics and produces a summary (both printed and saved to `eval/results.json`). The Q&A pairs are about `sample_docs`, so the harness always runs against `default_config/default_config.yaml` rather than your own `config.yaml`.\n\nIf some features are not working or missing, feel free to open an issue or a PR.\n\nMIT", "url": "https://wpnews.pro/news/show-hn-raggy-a-lightweight-cli-tool-for-rag-over-local-documents", "canonical_source": "https://github.com/paulknysh/raggy", "published_at": "2026-09-11 16:01:57+00:00", "updated_at": "2026-09-11 16:14:24.748867+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Raggy", "paulknysh", "LangChain", "Chroma", "Ollama", "GitHub", "OpenAI", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/show-hn-raggy-a-lightweight-cli-tool-for-rag-over-local-documents", "markdown": "https://wpnews.pro/news/show-hn-raggy-a-lightweight-cli-tool-for-rag-over-local-documents.md", "text": "https://wpnews.pro/news/show-hn-raggy-a-lightweight-cli-tool-for-rag-over-local-documents.txt", "jsonld": "https://wpnews.pro/news/show-hn-raggy-a-lightweight-cli-tool-for-rag-over-local-documents.jsonld"}}