Docs-Reference: A Common Lisp RAG System for Local Models A new open-source Common Lisp library, docs-reference, provides a local RAG (retrieval-augmented generation) system for referencing URL documentation with local AI models. The library, available on GitHub, indexes documentation pages by chunking text and creating BM25, dense vector, and ColBERT embeddings, then fuses retrieval results using Reciprocal Rank Fusion (RRF). It supports features like query rewriting, HyDE, and AgenticRAG, and offers functions for background crawling, question answering, and conversation management. A library for referencing URL documentation with a local RAG system. Written in Common Lisp. Clone this repository into ~/quicklisp/local-projects/ , or another location where it can be found by ASDF: bash $ git clone https://github.com/skarnati20/docs-reference.git then load it into Lisp and switch into its package: ql:quickload :docs-reference in-package :docs-reference Register a documentation site. The crawl follows same-origin links and embeds every chunk, so it may take some time — therefore rla runs it in the background and returns a task: js rla "https://lispcookbook.github.io/cl-cookbook/" tasks ; = S TASK :NAME "index ..." :STATUS :RUNNING ... print-corpora ; once done: each page and its chunk count Then ask a question. ds retrieves, then answers with the local model: ds "How do I define a struct with defstruct?" A query makes several model calls, so it can be slow. dsa does the same thing in the background, which is usually what you want: dsa "How do I read a file line by line?" To see the retrieved passages on their own, without generating an answer: find-docs "hash table" :top-k 5 dc keeps a rolling conversation, so follow-ups can refer back to earlier turns. clear-chat starts a fresh session: dc "How do I append two lists?" dc "What about doing that for an array?" clear-chat Managing what is indexed: print-links ; registered base URLs, numbered remove-link-idx 0 ; drop one by that number clear-docs ; drop everything The first step in the application is fetching and saving content from the requested URL. When indexing URLs, the program chunks the text and processes its keywords through BM25, and creates a dense vector embedding, as well as a ColBERT embedding. These are stored in-memory and on-disk. BM25 https://en.wikipedia.org/wiki/Okapi BM25 is a ranking algorithm that uses inverse document frequency to find the relevant documents based off keywords for a given query. Dense vector embeddings https://www.pinecone.io/learn/series/nlp/dense-vector-embeddings-nlp/ are a popular way of capturing the meaning of a given text, allowing us to understand the geometric relationship between two texts. ColBERT https://arxiv.org/abs/2004.12832 is another approach for getting text similarity, but it uses a more fine-grained token approach that provides an improvement over dense vectors. When a user asks a question, we use a fusion of these methods to retrieve back n requested results. Each method creates its own ordering, and the Reciprocal Rank Fusion algorithm RRF https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion allows us to combine these results simply. In addition, the following features have also been added: - Rewriting queries into sub-queries for more effective retrieval HyDE https://arxiv.org/abs/2212.10496 - Embedding a hypothetical answer to the query to get closer to the neighborhood where the real answer lies AgenticRAG https://research.google/blog/unlocking-dependable-responses-with-gemini-enterprise-agent-platforms-agentic-rag/ - Using agents to figure out when document context is sufficient Every query entry point ds , dsa , dc , dca passes its options straight through to run-rag , so any of these can be added to any query: | Option | Default | Description | |---|---|---| :agentic | t | Run the agentic pipeline retrieve, judge sufficiency, refine . nil retrieves once and answers. | :max-iterations | 3 | How many retrieve/assess rounds the agentic pipeline may take. Agentic only. | :top-k | 10 | How many chunks are handed to the model as context. | :methods | :dense :bm25 :colbert | Retrievers to fuse with RRF. Accepts :all , one keyword, or a list of :dense , :bm25 , :colbert . | :use-rewrite | t | Decompose the question into 1-3 focused sub-queries before searching. | :use-hyde | t | Write a hypothetical answer passage and search with it as well | :rerank nil :rerank-start 0 0 . :rerank-end nil nil means the whole list.Retrieve with one method only, skipping the servers the others need: ds "How do I define a struct with defstruct?" :methods :dense ds "How do I define a struct with defstruct?" :methods ' :dense :bm25 Skip the agentic loop for a single fast retrieval: ds "How do I concatenate strings?" :agentic nil Turn off the query transforms, so the question is searched verbatim: ds "with-open-file" :use-rewrite nil :use-hyde nil Re-rank the plain pipeline, leaving the top two hits in place and looking thirty deep: ds "How do I run an external shell command?" :agentic nil :rerank t :rerank-start 2 :rerank-end 30 Widen the context and allow more refinement rounds: dsa "How does the condition system work?" :top-k 20 :max-iterations 5 Options work the same in a chat session: dc "How do I append two lists?" :methods :dense :agentic nil The following instructions assume you have both Ollama and llama.cpp installed. The library relies on Ollama to provide 1 the embedding model for document chunks, and 2 the local LLM used for answering queries. In the following example, we will use qwen3-embedding:0.6b for the embeddings model and qwen3:1.7b for the chat model. These are set by default, and this combination was found to be quite fast and effective. Make sure Ollama is not already running, and run the following: Pull the models ollama pull qwen3-embedding:0.6b ollama pull qwen3:1.7b Allow multiple models for parallelism OLLAMA MAX LOADED MODELS=2 OLLAMA NUM PARALLEL=4 ollama serve To change the models, you will want to change embedding-model-name or chat-model-name in ollama-embeddings.lisp or ollama-chat.lisp respectively. You must run a model like so on port 8081 : llama serve -hf LiquidAI/LFM2-ColBERT-350M-GGUF:Q8 0 \ --embeddings --pooling none --port 8081 You may change the model by serving a different one with the command above. If you would like to use a re-ranking method after relevant documents have been retrieved, you can host one as so on port 8080 : llama serve -hf gpustack/jina-reranker-v2-base-multilingual-GGUF:Q5 K M \ --rerank --pooling rank --port 8080 You may change the model by serving a different one with the command above. docs-reference.asd system definition, and the file load order package.lisp the DOCS-REFERENCE package and its exports docs-reference.lisp entry points: register-link, docs-search, docs-chat, ds/dsa/dc/dca rag.lisp the pipelines: agentic-rag, default-rag, run-rag, and the agent steps chat-session.lisp rolling conversation state, and rendering it as chat messages fetcher.lisp fetching a URL and extracting readable text from its HTML content-types.lisp deciding what a URL points at, so binary links are skipped store.lisp the chunk and corpus structs, chunking, tokenizing, and indexing search.lisp search-corpus / search-corpora: which retrievers to fuse scoring.lisp per-retriever rankings, and the RRF fusion that combines them bm25.lisp the BM25 lexical index vector.lisp dense vector arithmetic: dot product, cosine, centroid lemmatization.lisp Porter2 stemmer, applied to both indexed and query tokens colbert.lisp ColBERT late-interaction embeddings and MaxSim scoring cross-encoder.lisp cross-encoder re-ranking of retrieved chunks ollama-embeddings.lisp embedding requests to Ollama ollama-chat.lisp chat completion requests to Ollama helpers.lisp JSON and string helpers shared by the request builders local-storage.lisp the on-disk cache for page text and ColBERT embeddings stream.lisp background tasks and the worker pool eval.lisp the evaluation harness: profiles, metrics, and comparison evals/common-lisp/ a query set with known-good pages, and saved run results eval.lisp defines a small evaluation framework where you can determine the right set of parameters for your use-case or workflow. In the following example, we run it over a Common Lisp eval set found in evals/common-lisp/ : First register the site the eval set was written against, and wait for the index to finish: rla "https://lispcookbook.github.io/cl-cookbook/" tasks ; wait for :DONE Then run every profile in eval-profiles over the same queries: run-eval-profiles corpora "evals/common-lisp/queries.json" :top-k 5 :output-dir "evals/common-lisp/" Each profile writes a results-