A library for referencing URL documentation with a local RAG system. Written in Common Lisp.
Clone this repository into ~/quicklisp/local-projects/
, or another location where it can be found by ASDF:
$ git clone https://github.com/skarnati20/docs-reference.git
then load it into Lisp and switch into its package:
(ql:quickload :docs-reference)
(in-package :docs-reference)
Register a documentation site. The crawl follows same-origin links and embeds every chunk, so it may take some time — therefore rla
runs it in the background and returns a task:
(rla "https://lispcookbook.github.io/cl-cookbook/")
(tasks) ; => #S(TASK :NAME "index ..." :STATUS :RUNNING ...)
(print-corpora) ; once done: each page and its chunk count
Then ask a question. ds
retrieves, then answers with the local model:
(ds "How do I define a struct with defstruct?")
A query makes several model calls, so it can be slow. dsa
does the same thing in the background, which is usually what you want:
(dsa "How do I read a file line by line?")
To see the retrieved passages on their own, without generating an answer:
(find-docs "hash table" :top-k 5)
dc
keeps a rolling conversation, so follow-ups can refer back to earlier turns. clear-chat
starts a fresh session:
(dc "How do I append two lists?")
(dc "What about doing that for an array?")
(clear-chat)
Managing what is indexed:
(print-links) ; registered base URLs, numbered
(remove-link-idx 0) ; drop one by that number
(clear-docs) ; drop everything
The first step in the application is fetching and saving content from the requested URL. When indexing URLs, the program chunks the text and processes its keywords through BM25, and creates a dense vector embedding, as well as a ColBERT embedding. These are stored in-memory and on-disk.
BM25 is a ranking algorithm that uses inverse document frequency to find the relevant documents based off keywords for a given query.
Dense vector embeddings are a popular way of capturing the meaning of a given text, allowing us to understand the geometric relationship between two texts.
ColBERT is another approach for getting text similarity, but it uses a more fine-grained token approach that provides an improvement over dense vectors.
When a user asks a question, we use a fusion of these methods to retrieve back n
requested results. Each method creates its own ordering, and the Reciprocal Rank Fusion algorithm (RRF) allows us to combine these results simply.
In addition, the following features have also been added:
- Rewriting queries into sub-queries for more effective retrieval HyDE- Embedding a hypothetical answer to the query to get closer to the neighborhood where the real answer liesAgenticRAG- Using agents to figure out when document context is sufficient
Every query entry point (ds
, dsa
, dc
, dca
) passes its options straight through to run-rag
, so any of these can be added to any query:
| Option | Default | Description |
|---|---|---|
:agentic |
||
t |
||
Run the agentic pipeline (retrieve, judge sufficiency, refine). nil retrieves once and answers. |
||
:max-iterations |
||
3 |
||
| How many retrieve/assess rounds the agentic pipeline may take. Agentic only. | ||
:top-k |
||
10 |
||
| How many chunks are handed to the model as context. | ||
:methods |
||
(:dense :bm25 :colbert) |
||
Retrievers to fuse with RRF. Accepts :all , one keyword, or a list of :dense , :bm25 , :colbert . |
||
:use-rewrite |
||
t |
||
| Decompose the question into 1-3 focused sub-queries before searching. | ||
:use-hyde |
||
t |
||
| Write a hypothetical answer passage and search with it as well ( | ||
:rerank
nil
:rerank-start
0
0
.:rerank-end
nil
nil
means the whole list.Retrieve with one method only, skipping the servers the others need:
(ds "How do I define a struct with defstruct?" :methods :dense)
(ds "How do I define a struct with defstruct?" :methods '(:dense :bm25))
Skip the agentic loop for a single fast retrieval:
(ds "How do I concatenate strings?" :agentic nil)
Turn off the query transforms, so the question is searched verbatim:
(ds "with-open-file" :use-rewrite nil :use-hyde nil)
Re-rank the plain pipeline, leaving the top two hits in place and looking thirty deep:
(ds "How do I run an external shell command?"
:agentic nil :rerank t :rerank-start 2 :rerank-end 30)
Widen the context and allow more refinement rounds:
(dsa "How does the condition system work?" :top-k 20 :max-iterations 5)
Options work the same in a chat session:
(dc "How do I append two lists?" :methods :dense :agentic nil)
The following instructions assume you have both Ollama
and llama.cpp
installed.
The library relies on Ollama to provide 1) the embedding model for document chunks, and 2) the local LLM used for answering queries.
In the following example, we will use qwen3-embedding:0.6b
for the embeddings model and qwen3:1.7b
for the chat model. These are set by default, and this combination was found to be quite fast and effective.
Make sure Ollama is not already running, and run the following:
ollama pull qwen3-embedding:0.6b
ollama pull qwen3:1.7b
OLLAMA_MAX_LOADED_MODELS=2 OLLAMA_NUM_PARALLEL=4 ollama serve
To change the models, you will want to change *embedding-model-name*
or *chat-model-name*
in ollama-embeddings.lisp
or ollama-chat.lisp
respectively.
You must run a model like so on port 8081
:
llama serve -hf LiquidAI/LFM2-ColBERT-350M-GGUF:Q8_0 \
--embeddings --pooling none --port 8081
You may change the model by serving a different one with the command above.
If you would like to use a re-ranking method after relevant documents have been retrieved, you can host one as so on port 8080
:
llama serve -hf gpustack/jina-reranker-v2-base-multilingual-GGUF:Q5_K_M \
--rerank --pooling rank --port 8080
You may change the model by serving a different one with the command above.
docs-reference.asd system definition, and the file load order
package.lisp the DOCS-REFERENCE package and its exports
docs-reference.lisp entry points: register-link, docs-search, docs-chat, ds/dsa/dc/dca
rag.lisp the pipelines: agentic-rag, default-rag, run-rag, and the agent steps
chat-session.lisp rolling conversation state, and rendering it as chat messages
fetcher.lisp fetching a URL and extracting readable text from its HTML
content-types.lisp deciding what a URL points at, so binary links are skipped
store.lisp the chunk and corpus structs, chunking, tokenizing, and indexing
search.lisp search-corpus / search-corpora: which retrievers to fuse
scoring.lisp per-retriever rankings, and the RRF fusion that combines them
bm25.lisp the BM25 lexical index
vector.lisp dense vector arithmetic: dot product, cosine, centroid
lemmatization.lisp Porter2 stemmer, applied to both indexed and query tokens
colbert.lisp ColBERT late-interaction embeddings and MaxSim scoring
cross-encoder.lisp cross-encoder re-ranking of retrieved chunks
ollama-embeddings.lisp embedding requests to Ollama
ollama-chat.lisp chat completion requests to Ollama
helpers.lisp JSON and string helpers shared by the request builders
local-storage.lisp the on-disk cache for page text and ColBERT embeddings
stream.lisp background tasks and the worker pool
eval.lisp the evaluation harness: profiles, metrics, and comparison
evals/common-lisp/ a query set with known-good pages, and saved run results
eval.lisp
defines a small evaluation framework where you can determine the right set of parameters for your use-case or workflow. In the following example, we run it over a Common Lisp eval set found in evals/common-lisp/
:
First register the site the eval set was written against, and wait for the index to finish:
(rla "https://lispcookbook.github.io/cl-cookbook/")
(tasks) ; wait for :DONE
Then run every profile in *eval-profiles*
over the same queries:
(run-eval-profiles *corpora*
"evals/common-lisp/queries.json"
:top-k 5
:output-dir "evals/common-lisp/")
Each profile writes a results-<name>.json
report next to the query set, and a comparison table is printed at the end, best first:
=== Profile comparison (top-k 5, ranked by RECALL) ===
profile MRR Recall Hit exact-R concep-R
dense+hyde 0.917 0.917 0.917 1.000 0.875
dense-only 0.883 0.883 0.883 1.000 0.850
bm25-only 0.583 0.583 0.583 0.353 0.700
best by RECALL: dense+hyde (0.917)
To score a single configuration rather than the whole set, call run-eval
with the same options a query takes:
(run-eval *corpora*
"evals/common-lisp/queries.json"
:top-k 5
:methods '(:dense :bm25)
:use-hyde t
:run-notes "hybrid + hyde")
To compare your own configurations, bind *eval-profiles*
, or pass a list in directly. Each profile is a :name
plus the options run-eval
accepts:
(run-eval-profiles *corpora*
"evals/common-lisp/queries.json"
:top-k 5
:profiles '((:name "dense" :methods (:dense))
(:name "dense+hyde" :methods (:dense) :use-hyde t)
(:name "dense+rerank" :methods (:dense) :rerank t
:rerank-end 30)))
To evaluate a different corpus, write a queries.json
alongside a baseUrl
, where each query lists the pages that should answer it:
{
"corpus": "Common Lisp Cookbook",
"baseUrl": "https://lispcookbook.github.io/cl-cookbook/",
"queries": [
{"query": "How do I define a struct with defstruct?",
"pages": ["data-structures.html"],
"type": "exact"}
]
}
A query counts as answered when a retrieved chunk comes from one of its pages
. The type
field is a free-form label, and is what splits the comparison table into its exact-R
and concep-R
columns.
I would like to thank Mark Watson, whose book Loving Common Lisp, or the Savvy Programmer's Secret Weapon inspired me to start my Lisp journey. This project drew inspiration from many of the ones included in the book. Support him by purchasing the book here.