Attention-Weighted Memory Graphs

A new LLM memory system called Attention-Weighted Memory Graphs uses Jina AI's late chunking to embed entire documents, then derives edge weights from the attention matrix averaged across layers and heads. Retrieval employs a budgeted breadth-first traversal where each hop consumes a fixed step cost plus a penalty proportional to edge weakness, and embeddings are updated via a weighted average that preserves document-level context.

I just had a new idea for how an LLM’s memory system could be structured. Roughly speaking, you can visualize it as a Graph-RAG setup where an entry point is located via an embedding, followed by step-by-step traversal along the edges. Late chunking The system builds on late chunking a technique introduced by Jina AI : rather than embedding each sentence independently, the whole document is tokenized and embedded first, and chunk-level vectors are derived afterward. This preserves document-level context inside each chunk’s embedding.In the Jina-embedding model the attention in the Transformer is calculated between sentences and not just tokens what I used inthememory. Attention defines the graph’s edges Instead of asking an LLM toextract nodes and edges for the graph,Jinaembeds an entire document at once and runs a forward pass withoutput attentions=True. The resulting attention matrix is then averaged across layers and heads, and used directly as the edge weight between chunks: how much did chunk A attend to chunk B while the model was processing the document as a whole? Swarming retrieval with a decaying budget The retrieval is starting from the node with the most similar embedding, the system does a breadth-first traversal where each hop consumes part of a starting “budget” - a combination of a fixed step cost and a penalty proportional to how weak the edge is. Traversal stops naturally once the budget runs out, rather than at an arbitrary hop limit or a fixed result count. The found nodes are put into the new document with the new entry and are the context for the new entry. Embedding updates When new information touches an existing node reached through swarming, the old embedding isn’t discarded or replaced. It’s nudged:updated = old embedding + alpha remaining budget new embedding, renormalized back to unit length. Nodes reached with more remaining budget i.e., more central to the current context get updated more; distant, weakly-connected nodes barely move .