Building a Persistent Codebase Memory System: Architecture, Hybrid Graph RAG, and Lessons Learned

Engineers Anish Pathak and Ambarish Pathak built a zero-token local codebase intelligence pipeline using deterministic AST parsing, LanceDB, and the Model Context Protocol (MCP), reducing ingestion time from 4 minutes to under 2 seconds with zero API cost. The architecture decouples repository ingestion from context querying to eliminate the 'context tax' of stateless AI coding assistants, and includes a custom local embedding pipeline to avoid silent failures when bypassing LLM entity extraction.

By Anish Pathak & Ambarish Pathak An architectural deep dive into designing a zero-token local codebase intelligence pipeline using deterministic AST parsing, LanceDB, and the Model Context Protocol MCP . AI coding assistants have fundamentally transformed developer productivity. They can write boilerplate, debug stack traces, refactor functions, and explain complex algorithmic flows in seconds. However, existing coding assistants suffer from a major architectural limitation: statelessness . Every time a developer opens a new terminal session or IDE window, the model starts from absolute zero. Developers are forced to repeatedly supply the same architectural overview, re-explain database schemas, and highlight shared utilities. This creates what we define as the context tax, the cumulative engineering time and cognitive load spent re-teaching an assistant information it should already retain. This problem escalates significantly with modern agentic terminal agents and IDEs such as Antigravity, Claude Code, Codex, and Cursor Agent Mode . When an autonomous agent is given a high-level task like “Refactor the authentication middleware” , the following execution loop occurs: To solve this, we engineered an architectural framework that decouples repository ingestion from context querying . The core design principle is simple: Repository ingestion should cost zero LLM tokens, run entirely locally, and construct a persistent graph-vector topology. Initial prototypes attempted to pass source files directly through an LLM to extract structural relationships and summaries. For a modest 30-file codebase, this consumed ~50,000 tokens during ingestion, hit free-tier rate limits within minutes, and took several minutes to process. We replaced this with deterministic Abstract Syntax Tree AST parsing : Deterministic Python AST extraction without LLM invocationimport astdef parse python ast content: str - dict: tree = ast.parse content symbols = {"classes": , "functions": , "imports": } for node in tree.body: if isinstance node, ast.ClassDef : bases = b.id for b in node.bases if isinstance b, ast.Name methods = {"name": item.name, "args": a.arg for a in item.args.args } for item in node.body if isinstance item, ast.FunctionDef, ast.AsyncFunctionDef symbols "classes" .append {"name": node.name, "bases": bases, "methods": methods} elif isinstance node, ast.FunctionDef, ast.AsyncFunctionDef : symbols "functions" .append {"name": node.name, "args": a.arg for a in node.args.args } return symbols Outcome : Ingestion time dropped from 4 minutes to under 2 seconds , with 100% deterministic symbol indexing and zero API cost. In hybrid graph-vector engines like Cognee, standard ingestion workflows typically link embedding generation with full LLM entity extraction cognify . Bypassing cognify to avoid LLM costs introduced a silent failure mode: raw files were stored in relational tables, but the vector collection DocumentChunk text was never initialized. Queries against the vector index returned empty results. To solve this, we engineered a dedicated local embedding pipeline fast cognify that executes document classification, semantic chunking, and local vector embedding while bypassing the LLM summarization task: python async def fast cognify dataset name: str : """ Minimal local embedding pipeline: Classify - Chunk - Local FastEmbed - LanceDB Storage Bypasses expensive LLM entity extraction. """ tasks = Task classify documents , Task extract chunks from documents, max chunk size=get max chunk tokens , chunker=TextChunker, , Task add data points, embed triplets=False, task config={"batch size": 100}, , await pipeline executor pipeline=run pipeline, tasks=tasks, datasets= dataset name , incremental loading=True, Outcome : Local 384-dimensional dense vector embeddings BAAI/bge-small-en-v1.5 are generated via ONNX Runtime on CPU, populating LanceDB in milliseconds without outbound API calls. When developers query the memory layer using cloud LLM providers, free-tier endpoints impose strict rate limits e.g., Requests Per Minute and Daily Token Limits . To ensure high availability without requiring paid enterprise tiers, we implemented a cyclic key rotation layer with automated cooldown tracking: python async def rotating acompletion args, kwargs : global last call time Enforce minimum call interval e.g., 4.5s to satisfy RPM caps async with rate limit lock: now = time.monotonic if now < last call time: sleep time = last call time + MIN CALL INTERVAL - now last call time += MIN CALL INTERVAL else: sleep time = 0.0 last call time = now + MIN CALL INTERVAL if sleep time 0: await asyncio.sleep sleep time Rotate to next active key filtering out keys on 429 cooldown kwargs 'api key' = get next active key return await original acompletion args, kwargs When an endpoint returns an HTTP 429 status code, that specific key is placed on a 10-minute cooldown window, and traffic routes seamlessly to alternative active keys. During early testing, queries occasionally experienced a 60-second latency spike before generating a response. Network tracing revealed that underlying LLM wrapper libraries such as LiteLLM attempt an outbound HTTPS request on initialization to fetch remote model pricing files from GitHub Raw. When executed behind corporate firewalls, VPNs, or strict Windows socket filters, the handshake timed out after 60 seconds before falling back to local defaults. Configuring local cost mapping resolved the bottleneck entirely: Force offline bundled model mapping; prevent outbound SSL handshake delaysos.environ "LITELLM LOCAL MODEL COST MAP" = "True"os.environ "LITELLM SUPPRESS PROVIDER INFO" = "True" Outcome : Query execution latency dropped from 60 seconds to under 3 seconds . In command-line applications, background tasks and unclosed client sessions frequently dump garbage-collection warnings to stderr upon process exit e.g., Unclosed client session, Task was destroyed but it is pending . Because libraries like aiohttp write directly to sys.stderr from their del destructor, standard warning filters warnings.filterwarnings cannot intercept them. We resolved this by adding an explicit graceful teardown routine prior to closing the asyncio event loop: python async def cleanup async resources loop : 1. Explicitly close singleton telemetry sessions if telemetry session and not telemetry session.closed: await telemetry session.close 2. Cancel and gather remaining fire-and-forget tasks pending = t for t in asyncio.all tasks loop if not t.done for task in pending: task.cancel await asyncio.gather pending, return exceptions=True To enable autonomous coding assistants such as Claude Code or Cursor to query codebase memory directly, we exposed the memory graph as a Model Context Protocol MCP server over standard stdio. python from fastmcp import FastMCPmcp = FastMCP "Codebase Memory Server" @mcp.toolasync def query codebase memory query: str - str: """ Query the persistent knowledge graph for architectural context, dependency graphs, or historical decision records. """ return await recall query query Instead of an AI agent performing 40 recursive filesystem tool calls to explore a project, it invokes query codebase memory once. It receives the exact top-3 relevant modules, architectural context, and related symbol definitions in a single 500-token response, cutting exploratory token consumption by over 90%. Building a local-first codebase memory engine highlighted several key software engineering principles: Building a Persistent Codebase Memory System: Architecture, Hybrid Graph RAG, and Lessons Learned https://pub.towardsai.net/building-a-persistent-codebase-memory-system-architecture-hybrid-graph-rag-and-lessons-learned-5d3dd3b90709 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.