{"slug": "show-hn-ragless-similar-to-rag-but-0-llm-api-costs-at-runtime", "title": "Show HN: RAGless – similar to RAG, but $0 LLM API costs at runtime", "summary": "RAGless, a retrieval-only question-answering system, eliminates LLM API costs at runtime by using vector retrieval with Gemini embeddings and a local Qdrant database, achieving zero hallucinations and near-zero cost per query. The project, consisting of three scripts for data preparation, ingestion, and a CLI chatbot, was released on Show HN.", "body_md": "**RAGless** is a retrieval-only question-answering system with **zero LLM calls at runtime**.\nSource documents are converted into self-contained informational blocks, indexed into a local vector database (Qdrant), and queried via asymmetric Gemini embeddings.\n\nZero hallucinations at runtime. Minimal latency. Near-zero cost per query.\n\nThe project consists of three independent scripts:\n\n| Script | Purpose |\n|---|---|\n`prepare_data.py` |\nExtracts Q&A blocks from source documents using Gemini in JSON mode |\n`ingest_to_qdrant.py` |\nGenerates embeddings and populates the local Qdrant vector database |\n`chatbot.py` |\nCLI chatbot that retrieves the most relevant answers for a user query |\n\n**No LLM at runtime**— The chatbot relies purely on vector retrieval. No expensive API calls during user interaction.** Robust Q-Q matching via**— If multiple question variants for the same answer match the query, their scores are summed. This makes the result far more stable than classic \"top-1\" retrieval.`answer_id`\n\naggregation**Asymmetric Gemini embeddings**—`RETRIEVAL_DOCUMENT`\n\nat ingestion time and`RETRIEVAL_QUERY`\n\nat retrieval time, as recommended by Google.**Embedded Qdrant**— No Docker server, no cloud service. Data is stored locally on disk (`./qdrant_data`\n\n).**Smart chunking**— Documents are chunked only if they exceed a token threshold, measured with the model's real tokenizer.** Optional Judge verification**—`prepare_data.py`\n\ncan enable a second LLM pass to discard blocks not supported by the source text (`--judge`\n\n).**Missed query logging**— Below-threshold queries are automatically logged to`missed_queries.log`\n\nfor later analysis.**Guaranteed idempotency**— Every run of`ingest_to_qdrant.py`\n\nrecreates the collection from scratch, preventing hidden duplicates.\n\n- Python 3.10+\n[Gemini API Key](https://aistudio.google.com/app/apikey)(free tier with generous limits)- Python dependencies (see Installation section)\n\n-\n**Clone or download the repository** and navigate to the project folder. -\n**Create a virtual environment**(recommended):\n\n```\npython -m venv venv\nsource venv/bin/activate  # Linux/macOS\n# or\nvenv\\Scripts\\activate   # Windows\n```\n\n-\n**Install dependencies**:\n\n```\npip install litellm qdrant-client pypdf python-dotenv tqdm\n```\n\n-\n**Configure your API key**:\n\n```\ncp .env.example .env\n# Edit .env and insert your GEMINI_API_KEY\n.\n├── source/                    # Folder with source documents (.pdf, .txt, .md)\n├── config.py                  # Centralized configuration (models, thresholds, paths)\n├── prepare_data.py            # Script 1: Q&A block extraction\n├── ingest_to_qdrant.py        # Script 2: embedding and indexing\n├── chatbot.py                 # Script 3: CLI chatbot\n├── data.json                  # Output of prepare_data.py (validated blocks)\n├── qdrant_data/               # Local vector database (auto-created)\n├── failed_chunks/             # Chunks that failed to produce valid JSON (debug)\n└── missed_queries.log         # Log of below-threshold queries\n```\n\nPlace your documents in the `source/`\n\nfolder (supports `.pdf`\n\n, `.txt`\n\n, `.md`\n\n), then run:\n\n```\npython prepare_data.py\n```\n\nWith optional Judge verification (slower but more accurate):\n\n```\npython prepare_data.py --judge\n```\n\n**What it does:**\n\n- Reads each file and counts tokens.\n- If the document is short (≤ 10,000 tokens), sends it whole to the LLM; otherwise splits it into chunks.\n- Extracts JSON blocks with\n`answer`\n\n,`questions`\n\n,`category`\n\n,`source_quote`\n\n. - Validates blocks and saves them to\n`data.json`\n\n.\n\n```\npython ingest_to_qdrant.py\n```\n\n**What it does:**\n\n- Loads\n`data.json`\n\n. - \"Explodes\" each block into as many rows as its question variants.\n- Generates embeddings in batches via LiteLLM + Gemini.\n- Recreates the Qdrant collection and inserts vectors with deterministic UUID5s.\n\n```\npython chatbot.py\n```\n\nAvailable options:\n\n```\npython chatbot.py --threshold 0.75   # Change the minimum aggregated score threshold\npython chatbot.py --debug            # Show internal scores and aggregation table\n```\n\n**Interaction:**\n\n```\nYou> How does check-in work?\n[INFO] Found 2 relevant answers, showing top 1:\n──────────────────────────────────────────────────────────────────────\n--- Answer 1 (Pertinence: 1.85) ---\nCheck-in is available from 3:00 PM to 8:00 PM. If you arrive after 8:00 PM,\nplease contact reception in advance...\n──────────────────────────────────────────────────────────────────────\nSource: source/regulations.txt\n```\n\nType `exit`\n\n, `quit`\n\n, or `:q`\n\nto leave.\n\nThe core of the chatbot is **aggregation by answer_id**:\n\n- The user query is embedded with\n`task_type=RETRIEVAL_QUERY`\n\n. - Qdrant returns the\n`TOP_K_RETRIEVAL`\n\nmost similar points (questions). - Scores of points pointing to the same\n`answer_id`\n\nare**summed**. - A candidate is shown only if:\n- the aggregated score exceeds\n`DEFAULT_THRESHOLD`\n\n**OR** - the best single hit exceeds\n`SINGLE_HIT_THRESHOLD`\n\n**AND** the best single hit is > 0.68 (minimum quality)\n\n- the aggregated score exceeds\n\nThis mechanism makes the system robust: even if no single question variant is a perfect match, the sum of multiple weak matches on the same answer can make it emerge correctly.\n\nClassic RAG generates answers at runtime by retrieving context and prompting an LLM to synthesize a response. **RAGless eliminates the generative step entirely.** Answers are pre-generated during ingestion and retrieved verbatim at runtime.\n\n| Advantage | Explanation |\n|---|---|\nMuch simpler pipeline |\nNo prompt engineering for answer generation, no context window management, no output parsing. Just embed, search, return. |\nQ-Q matching is far more reliable |\nMatching query-to-question (Q-Q) is semantically easier and more robust than query-to-document-chunk (Q-D) or query-to-answer (Q-A). Multiple question variants per answer provide redundancy. |\nDeterminism |\nSame question, same answer, always. Behavior is reproducible and testable. |\nZero hallucinations at runtime |\nNo LLM generates answers at query time. Returned text is pre-generated and immutable. |\nZero cost per query |\nAfter ingestion, there are no API calls. Retrieval is purely local computation. |\nLow latency |\nQuery embedding + vector search. Milliseconds, not seconds. |\nVerifiable answers |\nEvery block has `source_quote` and `source_file` . Complete audit trail. |\nReproducible bugs |\nIf an answer is wrong, it is 100% wrong. Easy to find and fix. |\nFinite output surface |\nThe number of possible answers is known (`data.json` ). Exhaustively testable. |\nRuns on modest hardware |\nEmbedded Qdrant, no LLM in memory. Works on CPU with a few GB of RAM. |\nTotal privacy |\nNo user data leaves the machine after ingestion. |\nNo dependency drift |\nThe embedding model can change, but the answers do not. You are not tied to an LLM provider's availability or pricing. |\n\n| Limitation | Explanation |\n|---|---|\nNo real-time flexibility |\nCannot synthesize novel answers, combine information across blocks, or adapt tone dynamically. What you ingest is what you get. |\nHigher ingestion cost |\nUsing an LLM to generate Q&A blocks costs more than simple embedding. See cost comparison below. |\nCoverage bounded by ingestion |\nIf a topic was not extracted during `prepare_data.py` , the system cannot answer it. No \"reasoning\" around gaps. |\nMaintenance requires re-ingestion |\nUpdating answers requires re-running the full pipeline, not just editing a prompt. |\n\n| Metric | Classic Generative RAG | Q-Q System (This Project) | Difference / Advantage |\n|---|---|---|---|\nIngestion Cost (One-time) |\n~$0.01 (embedding 100,000 tokens only) |\n~$1.50 (LLM generates Q&A from 100,000 tokens) |\n+$1.49 (Initial disadvantage, but negligible cost) |\nRuntime API Cost (Monthly) |\n~$157.50 (1,000 queries/day, ~1,000 context tokens + 150 output tokens) |\n$0.00 (only embedding 1,000 short queries, < $0.02/month) |\n~$157.50/month saved (Zero-cost scalability) |\nLatency (per query) |\n2.5 – 4 seconds (LLM text generation) |\n~0.15 seconds (pure vector search) |\n>15x faster (Instant response) |\nHallucination Rate (Runtime) |\nLow, but always > 0% | 0% |\nRisk eliminated (Static, deterministic output) |\n\nRAGless eliminates hallucinations **at runtime**, where they are most dangerous because they are uncontrollable. The risk during ingestion still exists, but it is mitigated — and crucially — **it happens offline, in a controlled environment, with the possibility of human review** before the knowledge base goes to production.\n\nRAGless shifts the hallucination risk from runtime to ingestion. Generation happens during `prepare_data.py`\n\n, which is why the optional `--judge`\n\npass and deterministic UUIDs for idempotency were added.\n\nThe trade-off is:you give up real-time flexibility in exchange for offline verifiability. For high-risk domains, I prefer hallucinations I can catch in a log over ones I cannot predict.\n\nAll tunable constants are in `config.py`\n\n:\n\n| Parameter | Description | Default |\n|---|---|---|\n`LLM_MODEL` |\nGemini model for extraction and judge | `gemini/gemini-2.5-flash` |\n`EMBEDDING_MODEL` |\nEmbedding model | `gemini/gemini-embedding-001` |\n`VECTOR_SIZE` |\nVector dimension (Matryoshka) | `3072` |\n`MAX_TOKENS_DOC` |\nThreshold for sending whole document | `10_000` |\n`CHUNK_SIZE` / `OVERLAP` |\nChunk size and overlap | `8_000` / `500` |\n`TOP_K_RETRIEVAL` |\nCandidates retrieved from Qdrant | `10` |\n`DEFAULT_THRESHOLD` |\nMinimum aggregated score threshold | `1.35` |\n`SINGLE_HIT_THRESHOLD` |\nFallback threshold on best single hit | `0.75` |\n`EMBEDDING_BATCH_SIZE` |\nQuestions per embedding API call | `100` |\n`QDRANT_UPSERT_BATCH` |\nPoints per upsert batch | `256` |\n\n| Problem | Solution |\n|---|---|\n`GEMINI_API_KEY not found` |\nCreate `.env` file with `GEMINI_API_KEY=...` |\n`Collection not found` |\nRun `python ingest_to_qdrant.py` first |\n| Malformed JSON in chunks | Check the `failed_chunks/` folder for raw text |\n| Empty LLM response | Possible Gemini safety block; try reducing or modifying source text |\n| Qdrant lockfile | Client closes automatically; in case of crash, manually remove the lock in `qdrant_data/` |\n\n**LiteLLM** is used as a unified proxy to call Gemini for both completions and embeddings.**Qdrant Client** in`path=`\n\nmode stores everything in local files: no server process is required.- Qdrant point IDs are deterministic UUID5s (\n`uuid5(NAMESPACE_DNS, answer_id:question_text)`\n\n), so re-running ingestion does not create logical duplicates.\n\nThis project is licensed under the **GNU Affero General Public License v3.0 (AGPLv3)**.\n\nSee [LICENSE](/EmilResearch/RAGless/blob/main/LICENSE) for details.", "url": "https://wpnews.pro/news/show-hn-ragless-similar-to-rag-but-0-llm-api-costs-at-runtime", "canonical_source": "https://github.com/EmilResearch/RAGless", "published_at": "2026-08-14 09:51:11+00:00", "updated_at": "2026-08-14 10:12:26.049048+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-infrastructure"], "entities": ["RAGless", "Gemini", "Qdrant", "LiteLLM", "Google"], "alternates": {"html": "https://wpnews.pro/news/show-hn-ragless-similar-to-rag-but-0-llm-api-costs-at-runtime", "markdown": "https://wpnews.pro/news/show-hn-ragless-similar-to-rag-but-0-llm-api-costs-at-runtime.md", "text": "https://wpnews.pro/news/show-hn-ragless-similar-to-rag-but-0-llm-api-costs-at-runtime.txt", "jsonld": "https://wpnews.pro/news/show-hn-ragless-similar-to-rag-but-0-llm-api-costs-at-runtime.jsonld"}}