Adding RAG: store your own judgment and grow your AI A developer built a per-message RAG "memory recall" hook for Claude Code that embeds past notes and conversations with BAAI's bge-m3 model and injects the closest matches into context before each reply, capped at 2,048 bytes. The hook splits long notes into 700-character chunks and scores each note by its best chunk, so semantic search surfaces notes written in Japanese for English queries that keyword search misses. Testing showed the highest-scoring unrelated question (0.597) outscored the median relevant one (0.589), complicating the relevance threshold. This is part 1 of the series "Adopting RAG — Grow your AI on your own accumulated judgment." Series contents My Claude Code setup loaded a 32KB index of its own notes at the start of every session, and it still kept making the mistakes those notes were written to prevent. "Take the screenshots for the English version in the English UI." "This article is too long." Each correction was saved as a memory note the first time I made it. One of those notes still quotes me asking how many times I have to say it. The notes were there. What was missing was the moment. When the situation came up mid-task, the AI did not go back and look. That is not specific to Claude Code: any AI coding agent with saved notes, Codex included, has to decide to search them, and that decision is the step that gets skipped. So I added RAG. RAG retrieval-augmented generation means that before the AI answers, something searches your own records and hands it the closest ones. Mine runs on every message I send: it searches my past notes and conversations by meaning and hands the closest few to the AI before it answers. We call it memory recall. It is part of ELN workflow, the in-house plugin suite we use with Claude Code. It relies on a Claude Code hook a script Claude Code runs automatically at fixed moments, here every time you send a message , so today it runs on Claude Code only. The idea would work for any agent. Nothing gets retrained. What changes is what the AI is handed. Every decision and correction I write down becomes something it can be shown the next time the situation comes up, so the more of my own judgment I store, the more the AI has to work with. That is what I mean by growing the AI with RAG. An earlier article covered a related hook that hands the AI a list of our architecture decisions once, at session start The real reason AI ignores existing decisions https://dev.to/uehara/the-real-reason-ai-ignores-existing-decisions-designing-an-adr-context-auto-injection-hook-34e2 . Resending the same list on every message was wasteful, so that hook moved to session start. This one works per message: what it hands over changes with what you just said, it hands over nothing or a short list when the match is weak, and it never exceeds 2,048 bytes. The screens below are real output from the live index on my Mac. The tool's own injection text is in Japanese only, so for this version I asked English questions and used jq a command-line JSON filter to print just the fields that matter. The command lines drop the script's install path, and the home folder in the output is shown as ~/. Here is the idea in one screen. I ask, in English, when the firm was founded. Searching the notes folder for "firm" or "founded" as whole words finds nothing. Searching by meaning puts the note about my name and company first, well ahead of the next one. That note records the founding year in Japanese, as 創業 2000年 "founded in 2000" . Neither word from the question appears anywhere in that note. The note does contain some English, such as the company name and the file name, but the part about the founding year is written only in Japanese. A text search can only find the exact characters. The search here turns both the question and every note into embeddings lists of numbers arranged so that texts with similar meaning end up close together and compares those. The model is bge-m3, published by BAAI, which handles many languages, including English and Japanese, in the same space. Long notes are split into 700-character pieces, and a note scores by its best piece, so that a detail further down than the first few lines still has a chance to be found. Handing notes to the AI on every message is only useful if it holds back when nothing is relevant. Otherwise the context fills with noise. This was the part that surprised me. In testing, the highest-scoring unrelated question scored 0.597. The median question that did have a correct note scored 0.589. Any cutoff on the score would either inject notes into unrelated messages or miss real ones. What did separate them was the shape of the results. When a message really matches a note, the top hit stands out. When it matches nothing, everything scores about the same. So the rule looks at the gap between the top score and the average of ranks 2 to 11, measured over notes and shared rules only past conversations shrink the gap . Across the unrelated questions that gap was at most 0.070. For questions with a correct note, the median gap was 0.080 to 0.096. | Gap between the top hit and the rest | What gets injected | |---|---| | Below 0.025, or a top score below 0.5 | Nothing | | 0.025 to 0.11 | Up to 3 notes or rules, 1,024 bytes | | 0.11 and above | Up to 3 notes or rules close to the top, plus up to 2 past conversations only when one clearly stands out among them 5 items, 2,048 bytes | Either way, a note from the current project that scores close to the top goes in first, so a similar shared rule cannot crowd it out. Messages shorter than 8 characters, like "go on", are not searched at all. Here is that selection applied to two messages: one asking why a Facebook page post did not show up, which my notes cover, and one about dinner. The Facebook question has a middling gap, so three notes go in. For the dinner question, the gap alone 0.041 would allow a short list, but the best note or rule scores only 0.396, below 0.5, so nothing goes in. In the real hook output, each item is one line: the kind memory note, shared rule or conversation , its name, the short description from the top of the note, and where the file is. The header line says these are possibly related past records, for reference and not instructions, and that the AI should read the file to check. The AI opens the file only if it decides the note is relevant. I checked it before building it. On a prototype against the 97 notes in my blog project, an AI agent with no context on the work wrote paraphrased questions, and I counted how often the right note came back. | Question set | Right note in top 3 | Right note first | |---|---|---| | The 3 questions in my original request | 3 of 3 | 2 of 3 | | 20 questions from an independent agent | 19 of 20 | 17 of 20 | | 20 more, held back until the end | 19 of 20 | 18 of 20 | The semantic code-search tool we already used semble put the three original questions at rank 3, rank 2 and nowhere. It is built for code, not for paraphrased Japanese. In the acceptance tests on the finished system, four fresh sets of 20 paraphrased questions put the right note in the top three for 15 to 19 of 20, and first for 9 to 15. The code-search tool got 1 or 2 first, in the three sets where it was measured. Choosing by similarity means a loosely related note sometimes gets in and a relevant one sometimes does not. Nothing is injected only when the gap is clearly small or the top score is below 0.5, as with the dinner question; in the final acceptance test, 17 of 20 unrelated messages still got a short list 1,020 bytes at most . So rules that must always hold are not left to recall. In my blog, a pre-publish check script blocks banned words and too few images mechanically. Recall is there to help the AI remember, not to enforce. Building the index means turning every note and every past conversation into embeddings. Doing that through an outside API would cost money on every call and would send all of those records to another company's service. So the embedding, the index and the search all run on the Mac, and no paid API is involved. Speed decided the shape. On this Mac Apple M3 Max, 128GB , keeping the model loaded on the GPU embeds a question in 43 to 116 ms. The embedding endpoint of ollama, a common way to run models locally, took 2.5 to 3.3 seconds per question, which is far too slow to wait for on every message. A reranker a second model that re-scores candidates took about 3 seconds for 10 items and gave the unrelated message "go on" a score of 0.82, so it could not tell relevant from irrelevant either. So a search process stays resident with the model in memory, and the hook asks it over a local socket. Those counts are index rows; long notes become several rows. Counted as files when I took the screenshots, the index held 1,964 memory notes across all my projects, 1,488 shared-rule files, and 3,585 conversation exchanges from 295 conversation logs. The index file is 84MB. How it keeps things local: There is also an optional team setting that shares notes and conversation records between our machines. A machine with it configured exchanges records, with secrets masked, only with a server on our office network, and never sends what you type. This Mac does not have it configured, so the index and numbers shown here are this Mac's alone. And it never blocks. The hook gives the search 300 ms. If there is no answer, the message goes through with nothing injected. If the search process is down, the conversation carries on. After 6 hours without a question, the process exits on its own. In the acceptance tests, the hook's 95th percentile once warm was 120 ms, and 248 ms with 10 simultaneous queries. When I took the screenshots, I sent the hook's query 60 more times: median 69 ms, 95th percentile 103 ms, worst 228 ms, none over 300 ms. It runs only for people. Most of our Claude Code sessions are unattended automation: of 3,734 conversation logs on the machine when I built this, 3,450 were automated runs and 279 were interactive. Those automated runs are neither searched for nor indexed. Before I built it, the notes for my blog project alone were 98 files and 600KB. Claude Code reads the note index a file called MEMORY.md, one line per note at the start of every session, and that index alone was 32KB against a target of 8KB. Three things were wrong: What I needed was for the right notes to appear on their own, on every message, even when worded differently, without loading more up front. Only the current project's notes, conversations and rules, plus the shared rules, are injected automatically. Notes from other projects would be confusing in the wrong place. The one exception is the team setting: on a machine with it configured, records synced from other machines through that setting can also be injected automatically, whichever project they come from. When I do want to know how another project handled something, recall.py query --repo