# Attention-Weighted Memory Graphs > Source: > Published: 2026-07-28 18:57:43+00:00 I just had a new idea for how an LLM’s memory system could be structured. Roughly speaking, you can visualize it as a Graph-RAG setup where an entry point is located via an embedding, followed by step-by-step traversal along the edges. **Late chunking** The system builds on*late chunking*(a technique introduced by Jina AI): rather than embedding each sentence independently, the whole document is tokenized and embedded first, and chunk-level vectors are derived afterward. This preserves document-level context inside each chunk’s embedding.In the Jina-embedding model the attention in the Transformer is calculated between sentences and not just tokens what I used inthememory. **Attention defines the graph’s edges** Instead of asking an LLM toextract nodes and edges for the graph,Jinaembeds an entire document at once and runs a forward pass withoutput_attentions=True. The resulting attention matrix is then averaged across layers and heads, and used directly as the edge weight between chunks: how much did chunk A attend to chunk B while the model was processing the document as a whole? **Swarming retrieval with a decaying budget** The retrieval is starting from the node with the most similar embedding, the system does a breadth-first traversal where each hop consumes part of a starting “budget” - a combination of a fixed step cost and a penalty proportional to how weak the edge is. Traversal stops naturally once the budget runs out, rather than at an arbitrary hop limit or a fixed result count. The found nodes are put into the new document with the new entry and are the context for the new entry. **Embedding updates** When new information touches an existing node reached through swarming, the old embedding isn’t discarded or replaced. It’s nudged:updated = old_embedding + alpha * remaining_budget * new_embedding, renormalized back to unit length. Nodes reached with more remaining budget (i.e., more central to the current context) get updated more; distant, weakly-connected nodes barely move .