{"slug": "i-made-my-ai-agent-s-notes-vault-retrievable-35-mb-embedder-llama-cpp-reranker", "title": "I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests", "summary": "A developer built Haystack, an open-source (MIT) semantic router for markdown note vaults that lets AI agents retrieve notes through hybrid search combining embeddings with corpus-IDF term statistics, RRF fusion, and cross-encoder reranking via llama.cpp, all running locally on a single 35 MB model at roughly 10 ms per query. The system turns user corrections into permanent rules and QA pairs that reindex and rerun as regression tests, exiting with code 1 on any regression, and it pins a learned phrasing only when it outranks the corpus's own best hit. The developer reports Recall@3 of 1.000 across an English vault, a Portuguese vault, and a foreign llama.cpp documentation corpus, and documented ten real failures in a shortcomings catalog.", "body_md": "Ok, I made a thing! LLMs have their... quirks. Namely re-reading the same notes, wandering back and forth, and spending 10x more tokens than needed because I can't be explicit.\n\nSo I built **Haystack** — a semantic router for a markdown vault. Instead of the agent digging through files, it queries: hybrid retrieval (embeddings + corpus-IDF term stats, RRF-fused, cross-encoder reranking via llama.cpp), all local, zero cloud, one 35 MB model, ~10 ms per query. MIT, one-command setup: [github.com/diogocccarvalho/haystack](https://github.com/diogocccarvalho/haystack).\n\n**Your corrections become regression tests.** Every retrieval miss you fix becomes a permanent rule AND a QA pair. The audit reindexes and runs every pair — it exits 1 on a regression, so the same question can't silently fail twice.\n\nAnd the learned rules only pin when they actually beat retrieval: a stored phrasing must outrank the corpus' own best hit for the same query. Scale-free, not absolute-threshold — so swapping embedders doesn't break your corrections. I A/B'd models against frozen QA sets and rejected two (a reranker that was 2x faster but -16pp Recall@3; a 97M embedder at -7pp). Numbers in [docs/RESEARCH.md](https://github.com/diogocccarvalho/haystack/blob/main/docs/RESEARCH.md).\n\n| Corpus | Recall@3 | MRR | \n|---|---|---|\n| English vault (68 pairs, nickname-style) | 1.000 | 0.917 | \n| Portuguese vault (14 pairs, synthetic) | 1.000 | 1.000 | \n| Foreign corpus (llama.cpp docs, 33 pairs) | 1.000 | 0.970 | \n\nThe foreign-corpus row matters more than the home-vault row: the chunker generalizes beyond the corpus it was tuned on. Also: `llama-server` does both the embeddings (bge-small-en GGUF) and the reranking (`--reranking` flag on ms-marco MiniLM-L12) — one binary, tested with llama.cpp v1.0.0.\n\nThe agent that uses Haystack daily also built most of it with me. Every gate in the repo was run by the agent against its own index — that's how the setup-script bugs got caught. And [docs/SHORTCOMINGS.md](https://github.com/diogocccarvalho/haystack/blob/main/docs/SHORTCOMINGS.md) is the failure catalog: ten real failures (a reranker overriding user corrections live; a rebuild that once deleted every correction; an over-context chunk killing a whole batch), each one now a design rule.\n\n```\n./setup.sh --start          # model + config + embeddings server, one command\npython3 haystack.py reindex\npython3 haystack.py query \"that thing about the invoices\"\n```\n\nAny OpenAI-compatible endpoint works instead (Ollama, LM Studio, remote API). Python 3.11+, stdlib + numpy only — no framework, no vector DB daemon, no cloud.\n\nThe open question I'd genuinely love answers to: this was tuned entirely on my vault. I want to know how the default config holds up on other people's notes. If you run it, tell me what breaks.", "url": "https://wpnews.pro/news/i-made-my-ai-agent-s-notes-vault-retrievable-35-mb-embedder-llama-cpp-reranker", "canonical_source": "https://dev.to/diogocccarvalho/i-made-my-ai-agents-notes-vault-retrievable-35-mb-embedder-llamacpp-reranker-corrections-as-4k34", "published_at": "2026-11-13 22:06:33+00:00", "updated_at": "2026-11-13 22:21:30.697839+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-search", "mlops"], "entities": ["Haystack", "llama.cpp", "bge-small-en", "ms-marco MiniLM-L12", "Ollama", "LM Studio", "diogocccarvalho"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-made-my-ai-agent-s-notes-vault-retrievable-35-mb-embedder-llama-cpp-reranker", "markdown": "https://wpnews.pro/news/i-made-my-ai-agent-s-notes-vault-retrievable-35-mb-embedder-llama-cpp-reranker.md", "text": "https://wpnews.pro/news/i-made-my-ai-agent-s-notes-vault-retrievable-35-mb-embedder-llama-cpp-reranker.txt", "jsonld": "https://wpnews.pro/news/i-made-my-ai-agent-s-notes-vault-retrievable-35-mb-embedder-llama-cpp-reranker.jsonld"}}