{"slug": "attention-weighted-memory-graphs", "title": "Attention-Weighted Memory Graphs", "summary": "A new LLM memory system called Attention-Weighted Memory Graphs uses Jina AI's late chunking to embed entire documents, then derives edge weights from the attention matrix averaged across layers and heads. Retrieval employs a budgeted breadth-first traversal where each hop consumes a fixed step cost plus a penalty proportional to edge weakness, and embeddings are updated via a weighted average that preserves document-level context.", "body_md": "I just had a new idea for how an LLM’s memory system could be structured. Roughly speaking, you can visualize it as a Graph-RAG setup where an entry point is located via an embedding, followed by step-by-step traversal along the edges.\n\n**Late chunking**\n\nThe system builds on*late chunking*(a technique introduced by Jina AI): rather than embedding each sentence independently, the whole document is tokenized and embedded first, and chunk-level vectors are derived afterward. This preserves document-level context inside each chunk’s embedding.In the Jina-embedding model the attention in the Transformer is calculated between sentences and not just tokens what I used inthememory.\n\n**Attention defines the graph’s edges**\n\nInstead of asking an LLM toextract nodes and edges for the graph,Jinaembeds an entire document at once and runs a forward pass withoutput_attentions=True. The resulting attention matrix is then averaged across layers and heads, and used directly as the edge weight between chunks: how much did chunk A attend to chunk B while the model was processing the document as a whole?\n\n**Swarming retrieval with a decaying budget**\n\nThe retrieval is starting from the node with the most similar embedding, the system does a breadth-first traversal where each hop consumes part of a starting “budget” - a combination of a fixed step cost and a penalty proportional to how weak the edge is. Traversal stops naturally once the budget runs out, rather than at an arbitrary hop limit or a fixed result count. The found nodes are put into the new document with the new entry and are the context for the new entry.\n\n**Embedding updates**\n\nWhen new information touches an existing node reached through swarming, the old embedding isn’t discarded or replaced. It’s nudged:updated = old_embedding + alpha * remaining_budget * new_embedding, renormalized back to unit length. Nodes reached with more remaining budget (i.e., more central to the current context) get updated more; distant, weakly-connected nodes barely move .", "url": "https://wpnews.pro/news/attention-weighted-memory-graphs", "canonical_source": "https://discuss.huggingface.co/t/attention-weighted-memory-graphs/178271#post_1", "published_at": "2026-07-28 18:57:43+00:00", "updated_at": "2026-07-28 19:05:29.293642+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Jina AI", "Attention-Weighted Memory Graphs"], "alternates": {"html": "https://wpnews.pro/news/attention-weighted-memory-graphs", "markdown": "https://wpnews.pro/news/attention-weighted-memory-graphs.md", "text": "https://wpnews.pro/news/attention-weighted-memory-graphs.txt", "jsonld": "https://wpnews.pro/news/attention-weighted-memory-graphs.jsonld"}}