{"slug": "searching-your-whole-ai-chat-history-without-a-vector-database-approach-research", "title": "Searching your whole AI chat history without a vector database — approach, research, and results", "summary": "A developer built a local, dependency-free system for searching AI coding-assistant chat transcripts without a vector database, using compact per-chat extracts, IDF keyword ranking with first-message and term-frequency topical boosts, and model-driven query expansion to bridge vocabulary gaps. In a test run, the tool surfaced the correct conversations about a dead-letter-queue fix and returned a cited answer that explicitly flagged what the excerpts did not contain, with only the query expansion and final answer requiring model calls.", "body_md": "AI coding assistants (Claude Code, and similar) save every conversation as a transcript on disk. Over\nmonths that becomes a big, unsearchable pile: *\"how did we fix that dead-letter-queue issue three weeks\nago?\"* is effectively unanswerable. The obvious answer is \"embed everything into a vector DB\" — but that\nadds a model, a store, a key, and a network dependency. This is a write-up of a **local, dependency-free**\nalternative that gets most of the benefit, the research it draws on, and measured results versus the\nnaïve baseline.\n\nAssistant transcripts are large JSONL files (tens of MB each), keyed by opaque UUIDs, scattered across a projects directory. The content that matters — decisions, the exact command that fixed something — is buried under ~97% tool-call/metadata noise.\n\nThe naïve fix, **\"full fetch\": hand whole transcripts to the model**, is slow (multi-MB prompts),\nexpensive (cost scales with input), lossy (the model skims), and doesn't scale past one or two chats.\n\n1. **Find** the right conversation(s) among dozens — including ones that used*different words* than the question.\n2. **Compress** transcripts to signal without losing intricate detail (commands, IDs, decisions).\n3. **Rank** by genuine relevance — not size or recency alone; handle chats that*drifted* into a topic.\n4. **Answer** with citations and an honest \"I don't know\", not a hallucination.\n5. Stay **fast, cheap, offline, and dependency-free** (must run anywhere a shell does).\n\n| Task | Technique | Research basis | \n|---|---|---|\n| Compress | A **compact extract** per chat — message text + tool-call*inputs* + head/tail of long results + reasoning — cached, ~100× smaller than raw. | Mem0 — *extract salient content, don't store raw* | \n| Rank | **IDF keyword** +**first-message topical boost** +**term-frequency topicality** (many mentions = \"about it\" — catches mid-chat drift) +**recency × importance** tie-breakers. | Generative Agents — *recency × importance × relevance* ; RAG —*retrieve then generate* | \n| Find (vocabulary gap) | **Query expansion instead of embeddings** : one cheap model call expands`dead-letter queue` →`dlq, redrive, sqs, reprocess` , then feed those into the local keyword ranker. | see §5 | \n| Answer | Retrieve top-K, round-robin turn-selection into a small budget, **one** model call with strict*cite-or-say-\"CANNOT ANSWER\"* instructions. | MemGPT — *page in only what's needed* ; RAG | \n\n**The pipeline** (steps 1–4 are local/offline/instant; only step 5 calls a model):\ntokenize the question → build the corpus (skip throwaway sessions) → rank locally → extract + select the\nmost relevant turns into a small budget → one model call that answers with citations.\n\nA real run (identifiers lightly redacted). Note the query never used the words \"dlq\" or \"redrive\" — the\nsynonym expansion bridged the gap and surfaced the right chats, and the model cited each source and was\nhonest about what the excerpts did *not* contain:\n\n``` bash\n$ ccask \"how did we fix the dead-letter queue backup?\"\n  expanding query (synonyms)…\nSearching 23 chats for: fix, dead, letter, queue, backup\n  (+ synonyms: dlq, redrive, sqs, reprocess, poison, retry, unprocessed, lambda)…\nMost relevant: Payments Backend, Data Pipeline Ops, Release Runbook, Onboarding, Newsletter\n\nThe dead-letter queue was cleared by adding the missing routing-map entry, then redriving:\n • \"77 of 96 messages cleared after adding the <mapping> entry\" (from \"Payments Backend\").\n • Root cause was a mapping gap — the fix was to add the missing entry, then redrive the\n   DLQ (from \"Data Pipeline Ops\").\n\nCANNOT ANSWER (fully): the excerpts give the outcome (add mapping entry → redrive → 77/96\ncleared) but not the exact commands, or what happened to the remaining 19 messages.\n```\n\nEverything above the answer is local and instant (the `expanding query` line is one small call; the\n`Searching…`/` Most relevant` lines are `grep`); only the final answer is a second model call over the\n~5 most-relevant chats' excerpts.\n\n*Observed on a real ~23-conversation history on one machine; illustrative, not a formal benchmark.*\n\n| Metric | Full fetch (raw transcript → model) | This approach (retrieve → one call) | \n|---|---|---|\n| Input per chat | **~6.5 MB** raw JSONL | **~63 KB** compact extract (**~100× smaller** ) | \n| Chats considered | 1–2 (doesn't scale) | **all ~23** , ranked; top ~5 read | \n| Ranking cost | n/a (you pick manually) | **local `grep`, milliseconds** — no model | \n| Latency (search across chats) | **~200–280 s** pointed at raw transcripts | **tens of seconds** (one small call); repeats reuse cache | \n| Single-chat summary (large chat) | minutes | **~34 s** , bounded; instant when cached | \n| Vocabulary-mismatch recall | none (must reuse exact words) | **yes** (synonym expansion) | \n| Answer honesty | skims, can hallucinate | quotes exact IDs/commands, or says `CANNOT ANSWER:` | \n| Dependencies | — | **none** beyond a shell + the assistant CLI (optional`python3` ) | \n\n**Consistent with the published Mem0 result** on the LOCOMO long-conversation benchmark — structured\nretrieval vs. full-context gave **~90% token savings, ~91% lower p95 latency, and +26% answer quality**\n([2504.19413](https://arxiv.org/abs/2504.19413)). Same shape here: far less input, far lower latency, and\n*better* (specific, cited) answers than dumping raw transcripts.\n\nThe textbook way to close the vocabulary gap (`dead-letter queue` vs `DLQ redrive`) is embeddings + cosine\nsimilarity. Two realizations changed the approach:\n\n1. **Cosine similarity is trivial to write — the hard part is the vectors.** Cosine is ~8 lines of code.\nBut*semantic* vectors (where synonyms land close) require a trained embedding model; lexical/TF-IDF\nvectors you can build yourself carry the same vocabulary-mismatch blind spot as keyword search. (And\nsome providers — e.g. Anthropic — have no native embeddings API, so you'd add a third-party model,\na key, and network calls.)\n2. **You can borrow the model's semantic knowledge without vectors** — one query-expansion call turns the\nquestion into the words the transcript likely used, then the free local ranker does the rest.\n\nNet: meaning-based recall with **no vector store, no embedding model, no extra dependency**. True\nembeddings and hierarchical **RAPTOR** ([2401.18059](https://arxiv.org/abs/2401.18059)) summaries remain a\nsensible upgrade for *much* larger histories.\n\n- **MemGPT: LLMs as Operating Systems** — Packer et al. 2023 —[2310.08560](https://arxiv.org/abs/2310.08560)\n- **Generative Agents: Interactive Simulacra of Human Behavior** — Park et al. 2023 —[2304.03442](https://arxiv.org/abs/2304.03442)\n- **Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory** — Chhikara et al. 2025 —[2504.19413](https://arxiv.org/abs/2504.19413)\n- **RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval** — Sarthi et al. 2024 —[2401.18059](https://arxiv.org/abs/2401.18059)\n- **Retrieval-Augmented Generation (RAG)** — Lewis et al. 2020 —[2005.11401](https://arxiv.org/abs/2005.11401)\n- **A Survey on the Memory Mechanism of LLM-based Agents** — Zhang et al. 2024 —[2404.13501](https://arxiv.org/abs/2404.13501) ·[tracking repo](https://github.com/nuster1128/LLM_Agent_Memory_Survey)\n\nThis approach ships as the `ccask` command in **Claudius**, a shell toolkit for working with your Claude\nCode history by name — [https://github.com/SaxenaKartik/claudius](https://github.com/SaxenaKartik/claudius)\n\n```\nccask \"how did we fix the DLQ redrive?\"          # search ALL your chats (default), answered + cited\nccask -c \"Backend Changes\" \"what did we decide?\" # scope to a specific chat\nccask -c                                         # pick chat(s) from a menu, then type the question\n```\n\nTunables: `CCASK_TOPK` (chats read, default 5), `CCASK_EXPAND` (0 disables synonym expansion),\n`CCASK_W_RECENCY` / `CCASK_W_IMPORTANCE` / `CCASK_W_FREQ` / `CCASK_RECENCY_DAYS` (ranking weights).\nEverything runs locally except the single answering call.", "url": "https://wpnews.pro/news/searching-your-whole-ai-chat-history-without-a-vector-database-approach-research", "canonical_source": "https://gist.github.com/SaxenaKartik/64bedbca68511032e8172599e3e188ab", "published_at": "2026-08-29 13:48:31+00:00", "updated_at": "2026-09-18 07:54:08.260675+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "natural-language-processing"], "entities": ["Claude Code", "Mem0", "Generative Agents", "MemGPT", "SQS", "Lambda"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/searching-your-whole-ai-chat-history-without-a-vector-database-approach-research", "markdown": "https://wpnews.pro/news/searching-your-whole-ai-chat-history-without-a-vector-database-approach-research.md", "text": "https://wpnews.pro/news/searching-your-whole-ai-chat-history-without-a-vector-database-approach-research.txt", "jsonld": "https://wpnews.pro/news/searching-your-whole-ai-chat-history-without-a-vector-database-approach-research.jsonld"}}