I just had a new idea for how an LLM’s memory system could be structured. Roughly speaking, you can visualize it as a Graph-RAG setup where an entry point is located via an embedding, followed by step-by-step traversal along the edges.
Late chunking
The system builds onlate chunking(a technique introduced by Jina AI): rather than embedding each sentence independently, the whole document is tokenized and embedded first, and chunk-level vectors are derived afterward. This preserves document-level context inside each chunk’s embedding.In the Jina-embedding model the attention in the Transformer is calculated between sentences and not just tokens what I used inthememory.
Attention defines the graph’s edges
Instead of asking an LLM toextract nodes and edges for the graph,Jinaembeds an entire document at once and runs a forward pass withoutput_attentions=True. The resulting attention matrix is then averaged across layers and heads, and used directly as the edge weight between chunks: how much did chunk A attend to chunk B while the model was processing the document as a whole?
Swarming retrieval with a decaying budget
The retrieval is starting from the node with the most similar embedding, the system does a breadth-first traversal where each hop consumes part of a starting “budget” - a combination of a fixed step cost and a penalty proportional to how weak the edge is. Traversal stops naturally once the budget runs out, rather than at an arbitrary hop limit or a fixed result count. The found nodes are put into the new document with the new entry and are the context for the new entry.
Embedding updates
When new information touches an existing node reached through swarming, the old embedding isn’t discarded or replaced. It’s nudged:updated = old_embedding + alpha * remaining_budget * new_embedding, renormalized back to unit length. Nodes reached with more remaining budget (i.e., more central to the current context) get updated more; distant, weakly-connected nodes barely move .