{"slug": "docs-reference-a-common-lisp-rag-system-for-local-models", "title": "Docs-Reference: A Common Lisp RAG System for Local Models", "summary": "A new open-source Common Lisp library, docs-reference, provides a local RAG (retrieval-augmented generation) system for referencing URL documentation with local AI models. The library, available on GitHub, indexes documentation pages by chunking text and creating BM25, dense vector, and ColBERT embeddings, then fuses retrieval results using Reciprocal Rank Fusion (RRF). It supports features like query rewriting, HyDE, and AgenticRAG, and offers functions for background crawling, question answering, and conversation management.", "body_md": "A library for referencing URL documentation with a local RAG system. Written in Common Lisp.\n\nClone this repository into `~/quicklisp/local-projects/`\n\n, or another location where it can be found by ASDF:\n\n``` bash\n$ git clone https://github.com/skarnati20/docs-reference.git\n```\n\nthen load it into Lisp and switch into its package:\n\n```\n(ql:quickload :docs-reference)\n(in-package :docs-reference)\n```\n\nRegister a documentation site. The crawl follows same-origin links and embeds every chunk, so it may take some time — therefore `rla`\n\nruns it in the background and returns a task:\n\n``` js\n(rla \"https://lispcookbook.github.io/cl-cookbook/\")\n\n(tasks)          ; => #S(TASK :NAME \"index ...\" :STATUS :RUNNING ...)\n(print-corpora)  ; once done: each page and its chunk count\n```\n\nThen ask a question. `ds`\n\nretrieves, then answers with the local model:\n\n```\n(ds \"How do I define a struct with defstruct?\")\n```\n\nA query makes several model calls, so it can be slow. `dsa`\n\ndoes the same thing in the background, which is usually what you want:\n\n```\n(dsa \"How do I read a file line by line?\")\n```\n\nTo see the retrieved passages on their own, without generating an answer:\n\n```\n(find-docs \"hash table\" :top-k 5)\n```\n\n`dc`\n\nkeeps a rolling conversation, so follow-ups can refer back to earlier turns. `clear-chat`\n\nstarts a fresh session:\n\n```\n(dc \"How do I append two lists?\")\n(dc \"What about doing that for an array?\")\n(clear-chat)\n```\n\nManaging what is indexed:\n\n```\n(print-links)        ; registered base URLs, numbered\n(remove-link-idx 0)  ; drop one by that number\n(clear-docs)         ; drop everything\n```\n\nThe first step in the application is fetching and saving content from the requested URL. When indexing URLs, the program chunks the text and processes its keywords through BM25, and creates a dense vector embedding, as well as a ColBERT embedding. These are stored in-memory and on-disk.\n\n[BM25](https://en.wikipedia.org/wiki/Okapi_BM25) is a ranking algorithm that uses inverse document frequency to find the relevant documents based off keywords for a given query.\n\n[Dense vector embeddings](https://www.pinecone.io/learn/series/nlp/dense-vector-embeddings-nlp/) are a popular way of capturing the meaning of a given text, allowing us to understand the geometric relationship between two texts.\n\n[ColBERT](https://arxiv.org/abs/2004.12832) is another approach for getting text similarity, but it uses a more fine-grained token approach that provides an improvement over dense vectors.\n\nWhen a user asks a question, we use a fusion of these methods to retrieve back `n`\n\nrequested results. Each method creates its own ordering, and the [Reciprocal Rank Fusion algorithm (RRF)](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion) allows us to combine these results simply.\n\nIn addition, the following features have also been added:\n\n- Rewriting queries into sub-queries for more effective retrieval\n[HyDE](https://arxiv.org/abs/2212.10496)- Embedding a hypothetical answer to the query to get closer to the neighborhood where the real answer lies[AgenticRAG](https://research.google/blog/unlocking-dependable-responses-with-gemini-enterprise-agent-platforms-agentic-rag/)- Using agents to figure out when document context is sufficient\n\nEvery query entry point (`ds`\n\n, `dsa`\n\n, `dc`\n\n, `dca`\n\n) passes its options straight through to `run-rag`\n\n, so any of these can be added to any query:\n\n| Option | Default | Description |\n|---|---|---|\n`:agentic` |\n`t` |\nRun the agentic pipeline (retrieve, judge sufficiency, refine). `nil` retrieves once and answers. |\n`:max-iterations` |\n`3` |\nHow many retrieve/assess rounds the agentic pipeline may take. Agentic only. |\n`:top-k` |\n`10` |\nHow many chunks are handed to the model as context. |\n`:methods` |\n`(:dense :bm25 :colbert)` |\nRetrievers to fuse with RRF. Accepts `:all` , one keyword, or a list of `:dense` , `:bm25` , `:colbert` . |\n`:use-rewrite` |\n`t` |\nDecompose the question into 1-3 focused sub-queries before searching. |\n`:use-hyde` |\n`t` |\nWrite a hypothetical answer passage and search with it as well (\n|\n\n`:rerank`\n\n`nil`\n\n`:rerank-start`\n\n`0`\n\n`0`\n\n.`:rerank-end`\n\n`nil`\n\n`nil`\n\nmeans the whole list.Retrieve with one method only, skipping the servers the others need:\n\n```\n(ds \"How do I define a struct with defstruct?\" :methods :dense)\n(ds \"How do I define a struct with defstruct?\" :methods '(:dense :bm25))\n```\n\nSkip the agentic loop for a single fast retrieval:\n\n```\n(ds \"How do I concatenate strings?\" :agentic nil)\n```\n\nTurn off the query transforms, so the question is searched verbatim:\n\n```\n(ds \"with-open-file\" :use-rewrite nil :use-hyde nil)\n```\n\nRe-rank the plain pipeline, leaving the top two hits in place and looking thirty deep:\n\n```\n(ds \"How do I run an external shell command?\"\n    :agentic nil :rerank t :rerank-start 2 :rerank-end 30)\n```\n\nWiden the context and allow more refinement rounds:\n\n```\n(dsa \"How does the condition system work?\" :top-k 20 :max-iterations 5)\n```\n\nOptions work the same in a chat session:\n\n```\n(dc \"How do I append two lists?\" :methods :dense :agentic nil)\n```\n\nThe following instructions assume you have both `Ollama`\n\nand `llama.cpp`\n\ninstalled.\n\nThe library relies on Ollama to provide 1) the embedding model for document chunks, and 2) the local LLM used for answering queries.\n\nIn the following example, we will use `qwen3-embedding:0.6b`\n\nfor the embeddings model and `qwen3:1.7b`\n\nfor the chat model. These are set by default, and this combination was found to be quite fast and effective.\n\nMake sure Ollama is not already running, and run the following:\n\n```\n# Pull the models\nollama pull qwen3-embedding:0.6b\nollama pull qwen3:1.7b\n\n# Allow multiple models for parallelism\nOLLAMA_MAX_LOADED_MODELS=2 OLLAMA_NUM_PARALLEL=4 ollama serve\n```\n\nTo change the models, you will want to change `*embedding-model-name*`\n\nor `*chat-model-name*`\n\nin `ollama-embeddings.lisp`\n\nor `ollama-chat.lisp`\n\nrespectively.\n\nYou must run a model like so on port `8081`\n\n:\n\n```\nllama serve -hf LiquidAI/LFM2-ColBERT-350M-GGUF:Q8_0 \\\n  --embeddings --pooling none --port 8081\n```\n\nYou may change the model by serving a different one with the command above.\n\nIf you would like to use a re-ranking method after relevant documents have been retrieved, you can host one as so on port `8080`\n\n:\n\n```\nllama serve -hf gpustack/jina-reranker-v2-base-multilingual-GGUF:Q5_K_M \\\n  --rerank --pooling rank --port 8080\n```\n\nYou may change the model by serving a different one with the command above.\n\n```\ndocs-reference.asd        system definition, and the file load order\npackage.lisp              the DOCS-REFERENCE package and its exports\n\ndocs-reference.lisp       entry points: register-link, docs-search, docs-chat, ds/dsa/dc/dca\nrag.lisp                  the pipelines: agentic-rag, default-rag, run-rag, and the agent steps\nchat-session.lisp         rolling conversation state, and rendering it as chat messages\n\nfetcher.lisp              fetching a URL and extracting readable text from its HTML\ncontent-types.lisp        deciding what a URL points at, so binary links are skipped\nstore.lisp                the chunk and corpus structs, chunking, tokenizing, and indexing\n\nsearch.lisp               search-corpus / search-corpora: which retrievers to fuse\nscoring.lisp              per-retriever rankings, and the RRF fusion that combines them\nbm25.lisp                 the BM25 lexical index\nvector.lisp               dense vector arithmetic: dot product, cosine, centroid\nlemmatization.lisp        Porter2 stemmer, applied to both indexed and query tokens\ncolbert.lisp              ColBERT late-interaction embeddings and MaxSim scoring\ncross-encoder.lisp        cross-encoder re-ranking of retrieved chunks\n\nollama-embeddings.lisp    embedding requests to Ollama\nollama-chat.lisp          chat completion requests to Ollama\nhelpers.lisp              JSON and string helpers shared by the request builders\n\nlocal-storage.lisp        the on-disk cache for page text and ColBERT embeddings\nstream.lisp               background tasks and the worker pool\n\neval.lisp                 the evaluation harness: profiles, metrics, and comparison\nevals/common-lisp/        a query set with known-good pages, and saved run results\n```\n\n`eval.lisp`\n\ndefines a small evaluation framework where you can determine the right set of parameters for your use-case or workflow. In the following example, we run it over a Common Lisp eval set found in `evals/common-lisp/`\n\n:\n\nFirst register the site the eval set was written against, and wait for the index to finish:\n\n```\n(rla \"https://lispcookbook.github.io/cl-cookbook/\")\n(tasks)  ; wait for :DONE\n```\n\nThen run every profile in `*eval-profiles*`\n\nover the same queries:\n\n```\n(run-eval-profiles *corpora*\n                   \"evals/common-lisp/queries.json\"\n                   :top-k 5\n                   :output-dir \"evals/common-lisp/\")\n```\n\nEach profile writes a `results-<name>.json`\n\nreport next to the query set, and a comparison table is printed at the end, best first:\n\n```\n=== Profile comparison (top-k 5, ranked by RECALL) ===\n             profile       MRR    Recall       Hit   exact-R  concep-R\n          dense+hyde     0.917     0.917     0.917     1.000     0.875\n          dense-only     0.883     0.883     0.883     1.000     0.850\n           bm25-only     0.583     0.583     0.583     0.353     0.700\n\nbest by RECALL: dense+hyde (0.917)\n```\n\nTo score a single configuration rather than the whole set, call `run-eval`\n\nwith the same options a query takes:\n\n```\n(run-eval *corpora*\n          \"evals/common-lisp/queries.json\"\n          :top-k 5\n          :methods '(:dense :bm25)\n          :use-hyde t\n          :run-notes \"hybrid + hyde\")\n```\n\nTo compare your own configurations, bind `*eval-profiles*`\n\n, or pass a list in directly. Each profile is a `:name`\n\nplus the options `run-eval`\n\naccepts:\n\n```\n(run-eval-profiles *corpora*\n                   \"evals/common-lisp/queries.json\"\n                   :top-k 5\n                   :profiles '((:name \"dense\"        :methods (:dense))\n                               (:name \"dense+hyde\"   :methods (:dense) :use-hyde t)\n                               (:name \"dense+rerank\" :methods (:dense) :rerank t\n                                                     :rerank-end 30)))\n```\n\nTo evaluate a different corpus, write a `queries.json`\n\nalongside a `baseUrl`\n\n, where each query lists the pages that should answer it:\n\n```\n{\n  \"corpus\": \"Common Lisp Cookbook\",\n  \"baseUrl\": \"https://lispcookbook.github.io/cl-cookbook/\",\n  \"queries\": [\n    {\"query\": \"How do I define a struct with defstruct?\",\n     \"pages\": [\"data-structures.html\"],\n     \"type\": \"exact\"}\n  ]\n}\n```\n\nA query counts as answered when a retrieved chunk comes from one of its `pages`\n\n. The `type`\n\nfield is a free-form label, and is what splits the comparison table into its `exact-R`\n\nand `concep-R`\n\ncolumns.\n\nI would like to thank Mark Watson, whose book *Loving Common Lisp, or the Savvy Programmer's Secret Weapon* inspired me to start my Lisp journey. This project drew inspiration from many of the ones included in the book. Support him by purchasing the book [here](https://leanpub.com/lovinglisp).", "url": "https://wpnews.pro/news/docs-reference-a-common-lisp-rag-system-for-local-models", "canonical_source": "https://github.com/skarnati20/docs-reference", "published_at": "2026-08-27 02:04:02+00:00", "updated_at": "2026-08-27 02:18:18.035640+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-tools", "ai-research"], "entities": ["docs-reference", "Common Lisp", "GitHub", "BM25", "ColBERT", "Reciprocal Rank Fusion", "HyDE", "AgenticRAG"], "alternates": {"html": "https://wpnews.pro/news/docs-reference-a-common-lisp-rag-system-for-local-models", "markdown": "https://wpnews.pro/news/docs-reference-a-common-lisp-rag-system-for-local-models.md", "text": "https://wpnews.pro/news/docs-reference-a-common-lisp-rag-system-for-local-models.txt", "jsonld": "https://wpnews.pro/news/docs-reference-a-common-lisp-rag-system-for-local-models.jsonld"}}