Developing scalable and production ready agentic applications will invariably require external memory storage. LLMs are stateless. Without memory, an agent would be unable to recall anything from prior interactions. External memory fixes this problem, letting agents persist key facts and context and pull them back from earlier conversations and interactions when relevant.
Most current agentic development platforms offer some type of memory service, e.g., AWS has Agentcore Memory, GCP has Vertex AI Memory Bank. The Databricks suite of agentic memory offerings is continually evolving, with significant changes as of Sep 2026:
The rest of this post walks through best practices around organizing memory across Lakebase partitions, the different types of memory an agent may need, and strategies for writing and then retrieving that memory.
How well memory is organized in the memory store will determine how cleanly it isolates users and how efficiently agents can retrieve it. This section will go over the identifiers that do the memory isolation, because short term and long term memory (discussed next section) are really just different answers to which identifier keys the memory.
Getting these four terms straight is crucial. Definitions can vary depending on the agent framework being used, but the key insight here is that short term memory is scoped to individual sessions while long term memory is scoped to individual users.
The key insight: thread/session IDs partition memory horizontally by conversation while user IDs partition it vertically by owner. This is why a thread scoped short term memory field can't carry information across different conversations, and why long-term memory keys on the user instead.
Prior to this new release of managed memory, you would need to manually
define a traditional postgres schema to store memory in Lakebase postgres. Now, with managed memory store using Lakebase, you simply set a few inputs and Databricks will define the memory store container, very similar to AWS and Google’s managed memory services.
Inside a store, each memory is a single entry made up of:
actor_id is the user, session_id is the session it came from, and path is a just a custom file like name given to help identify the container.
It can be helpful to think of agentic memory at three different layers, from most ephemeral to most durable.
The most ephemeral layer of memory, and arguably the foundation the others build on, is the in-memory checkpointer (an in process state snapshot). Many modern agent frameworks expose this feature in some form. In LangGraph it’s explicitly the checkpointer that snapshots the graph’s state after every step (each node execution). The AWS Strands framework expresses this via Sessions Management and Claude Agents SDK utilizes session transcript.
In short, in memory checkpointers snapshot the history of messages and tool calls so an agent can , resume andretry a failed step OR rewind and reuse the same identifier (thread_id in LangGraph, session_id in Strands and the Claude Agent SDK).
The problem is that an in memory checkpointer holds all of that in RAM which is fine for local development, but untenable for production grade agentic apps. A durable backend allows it to survive restarts.
In the context of agentic memory, short-term memory is an agent’s in-process state (the history of messages and tool calls within one conversation) persisted to a durable external store rather than held only in RAM.
By definition, short-term memory is scoped to a single thread_id/session_id, and that is exactly what an in memory checkpointer captures. Typically, the agentic framework has some mechanism and built in module to take in memory sessions and persist it to external short term memory.
Where short-term memory is keyed by which conversation (thread_id/session_id), long-term memory is keyed by a user identity. It scopes to the user rather than the thread. That single change is what lets a fact outlive any one conversation: the agent extracts information, preferences, past decisions and stores it under the user’s identity, so a brand-new thread tomorrow can still recall it. Short-term memory remembers this conversation; long-term memory remembers this user.
To scope memory to a user, you need a user identity. This is where an Identity Provider (IdP) comes into play. The user authenticates against an IdP (Okta, AWS Cognito, etc) and the resulting OAuth / on behalf of (OBO) token carries a verified end-user identity.
Extracting memory from raw conversations and writing them to Databricks managed memory is a simple function call:
memory_store.add(actor_id="user-123",path="/preferences/communication.md",content="Prefers email over phone. Timezone: PST. Enterprise subscription.",description="User 123 communication preferences",)
The more challenging aspect of memory design is deciding what releavnt information from raw user interactions should be persisted and written to external memory. Ideally, agents should be guided to extract things that will be relevant about the user in the long term future for long term memory while ignoring irrelevant small talks or one off facts about the user.
Based on what has been released so far, Databricks managed memory ships no built in extraction strategies. Instead, you define custom extraction strategies to have agents extract what you think is relevant.
On the otherhand, AWS Agentcore Memory does come with some built in extraction strategies that can be a good starting point for building custom extraction strategies in Databricks.
In Agentcore, a strategy is essentially an extraction prompt the platform runs over the raw event stream: 1. Semantic pulls out durable facts, 2. Summary stores session scoped summary, 3. User Preferences captures how the user likes things, and 4. Episodic (the most complex extraction strategy) records specific past events.
For example, episodic memory would capture what happened in a given interaction (what decision was made, how a problem was solved or approached) so the agent can recall that episode later.
When using Databricks managed memory, you implement each of these yourself. Since you have full control, you can get as creative as you want when implementing the memory extraction in your code. There are several poossible ways of doing this:
When new user interactions are launched, the agent should pull out relevant information from the memory store and generate personalized responses based off that retrieved memory. In Databricks, this is done via natural language search:
#List memory entries memory_entries = memory_store.list(actor_id="user-123",path_prefix="/preferences/",)#Open and retrieve memory contentmemory_results = memory_store.search(actor_id="user-123", query="communication preferences", limit=10)
The Databricks docs indicate this is a ‘BM25 full-text relevance and returns a top-N set (up to 100 entries).’ It is not standard vector similarity search in the same way AWS Agentcore memory utilizes. Everything that is retrieved is injected into the model’s context window, so proper context window management requires prudent memory retrieval. Best practice is to run list first, to return all memory entries and search through those two determine specific entries needed by the current interaction.
As with everything else in the AI space, Databricks memory is continually evolving and improving. The initial Databricks solution to agentic memory was to stand up a fully self managed Lakebase Postgres DB. Then Databricks began introducing managed agentic memory using Unity Catalog, but it appears as of Sep 2026 their managed memory offering has switched from Unity Catalog to Lakebase.
Implement Agentic Memory with Databricks Lakebase was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.