# Building a Persistent Codebase Memory System: Architecture, Hybrid Graph RAG, and Lessons Learned

> Source: <https://pub.towardsai.net/building-a-persistent-codebase-memory-system-architecture-hybrid-graph-rag-and-lessons-learned-5d3dd3b90709?source=rss----98111c9905da---4>
> Published: 2026-08-12 12:31:01+00:00

*By Anish Pathak & Ambarish Pathak*

*An architectural deep dive into designing a zero-token local codebase intelligence pipeline using deterministic AST parsing, LanceDB, and the Model Context Protocol (MCP).*

AI coding assistants have fundamentally transformed developer productivity. They can write boilerplate, debug stack traces, refactor functions, and explain complex algorithmic flows in seconds.

However, existing coding assistants suffer from a major architectural limitation: **statelessness**.

Every time a developer opens a new terminal session or IDE window, the model starts from absolute zero. Developers are forced to repeatedly supply the same architectural overview, re-explain database schemas, and highlight shared utilities.

This creates what we define as the **context tax, **the cumulative engineering time and cognitive load spent re-teaching an assistant information it should already retain.

This problem escalates significantly with modern **agentic terminal agents and IDEs** (such as Antigravity, Claude Code, Codex, and Cursor Agent Mode).

When an autonomous agent is given a high-level task like *“Refactor the authentication middleware”*, the following execution loop occurs:

To solve this, we engineered an architectural framework that decouples **repository ingestion** from **context querying**.

The core design principle is simple: **Repository ingestion should cost zero LLM tokens, run entirely locally, and construct a persistent graph-vector topology.**

Initial prototypes attempted to pass source files directly through an LLM to extract structural relationships and summaries. For a modest 30-file codebase, this consumed ~50,000 tokens during ingestion, hit free-tier rate limits within minutes, and took several minutes to process.

We replaced this with **deterministic Abstract Syntax Tree (AST) parsing**:

```
# Deterministic Python AST extraction without LLM invocationimport astdef parse_python_ast(content: str) -> dict:    tree = ast.parse(content)    symbols = {"classes": [], "functions": [], "imports": []}        for node in tree.body:        if isinstance(node, ast.ClassDef):            bases = [b.id for b in node.bases if isinstance(b, ast.Name)]            methods = [                {"name": item.name, "args": [a.arg for a in item.args.args]}                for item in node.body if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef))            ]            symbols["classes"].append({"name": node.name, "bases": bases, "methods": methods})        elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):            symbols["functions"].append({"name": node.name, "args": [a.arg for a in node.args.args]})                return symbols
```

**Outcome**: Ingestion time dropped from 4 minutes to **under 2 seconds**, with 100% deterministic symbol indexing and zero API cost.

In hybrid graph-vector engines like Cognee, standard ingestion workflows typically link embedding generation with full LLM entity extraction (cognify()).

Bypassing cognify() to avoid LLM costs introduced a silent failure mode: raw files were stored in relational tables, but the vector collection (DocumentChunk_text) was never initialized. Queries against the vector index returned empty results.

To solve this, we engineered a dedicated local embedding pipeline (_fast_cognify) that executes document classification, semantic chunking, and local vector embedding while bypassing the LLM summarization task:

``` python
async def _fast_cognify(dataset_name: str):    """    Minimal local embedding pipeline:    Classify -> Chunk -> Local FastEmbed -> LanceDB Storage    Bypasses expensive LLM entity extraction.    """    tasks = [        Task(classify_documents),        Task(            extract_chunks_from_documents,            max_chunk_size=get_max_chunk_tokens(),            chunker=TextChunker,        ),        Task(            add_data_points,            embed_triplets=False,            task_config={"batch_size": 100},        ),    ]        await pipeline_executor(        pipeline=run_pipeline,        tasks=tasks,        datasets=[dataset_name],        incremental_loading=True,    )
```

**Outcome**: Local 384-dimensional dense vector embeddings (BAAI/bge-small-en-v1.5) are generated via ONNX Runtime on CPU, populating LanceDB in milliseconds without outbound API calls.

When developers query the memory layer using cloud LLM providers, free-tier endpoints impose strict rate limits (e.g., Requests Per Minute and Daily Token Limits).

To ensure high availability without requiring paid enterprise tiers, we implemented a cyclic key rotation layer with automated cooldown tracking:

``` python
async def _rotating_acompletion(*args, **kwargs):    global _last_call_time        # Enforce minimum call interval (e.g., 4.5s) to satisfy RPM caps    async with _rate_limit_lock:        now = time.monotonic()        if now < _last_call_time:            sleep_time = (_last_call_time + MIN_CALL_INTERVAL) - now            _last_call_time += MIN_CALL_INTERVAL        else:            sleep_time = 0.0            _last_call_time = now + MIN_CALL_INTERVAL                if sleep_time > 0:        await asyncio.sleep(sleep_time)            # Rotate to next active key (filtering out keys on 429 cooldown)    kwargs['api_key'] = get_next_active_key()    return await original_acompletion(*args, **kwargs)
```

When an endpoint returns an HTTP 429 status code, that specific key is placed on a 10-minute cooldown window, and traffic routes seamlessly to alternative active keys.

During early testing, queries occasionally experienced a 60-second latency spike before generating a response.

Network tracing revealed that underlying LLM wrapper libraries (such as LiteLLM) attempt an outbound HTTPS request on initialization to fetch remote model pricing files from GitHub Raw. When executed behind corporate firewalls, VPNs, or strict Windows socket filters, the handshake timed out after 60 seconds before falling back to local defaults.

Configuring local cost mapping resolved the bottleneck entirely:

```
# Force offline bundled model mapping; prevent outbound SSL handshake delaysos.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"os.environ["LITELLM_SUPPRESS_PROVIDER_INFO"] = "True"
```

**Outcome**: Query execution latency dropped from >60 seconds to **under 3 seconds**.

In command-line applications, background tasks and unclosed client sessions frequently dump garbage-collection warnings to stderr upon process exit (e.g., Unclosed client session, Task was destroyed but it is pending!).

Because libraries like aiohttp write directly to sys.stderr from their __del__ destructor, standard warning filters (warnings.filterwarnings) cannot intercept them.

We resolved this by adding an explicit graceful teardown routine prior to closing the asyncio event loop:

``` python
async def cleanup_async_resources(loop):    # 1. Explicitly close singleton telemetry sessions    if _telemetry_session and not _telemetry_session.closed:        await _telemetry_session.close()            # 2. Cancel and gather remaining fire-and-forget tasks    pending = [t for t in asyncio.all_tasks(loop) if not t.done()]    for task in pending:        task.cancel()    await asyncio.gather(*pending, return_exceptions=True)
```

To enable autonomous coding assistants (such as Claude Code or Cursor) to query codebase memory directly, we exposed the memory graph as a **Model Context Protocol (MCP)** server over standard stdio.

``` python
from fastmcp import FastMCPmcp = FastMCP("Codebase Memory Server")@mcp.toolasync def query_codebase_memory(query: str) -> str:    """    Query the persistent knowledge graph for architectural context,    dependency graphs, or historical decision records.    """    return await recall_query(query)
```

Instead of an AI agent performing 40 recursive filesystem tool calls to explore a project, it invokes query_codebase_memory() once. It receives the exact top-3 relevant modules, architectural context, and related symbol definitions in a single 500-token response, cutting exploratory token consumption by over 90%.

Building a local-first codebase memory engine highlighted several key software engineering principles:

[Building a Persistent Codebase Memory System: Architecture, Hybrid Graph RAG, and Lessons Learned](https://pub.towardsai.net/building-a-persistent-codebase-memory-system-architecture-hybrid-graph-rag-and-lessons-learned-5d3dd3b90709) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.