How I Implemented Memory in my AI Agent REXA

A developer implemented a long-term memory pipeline for the REXA AI agent, enabling users to explicitly save preferences via a save_memory tool. The CLI sends memory text through an authenticated HTTP API to the REXA website backend, which verifies the bearer token, generates embeddings, and stores entries in PostgreSQL with pgvector. Memory retrieval is not yet implemented; only the saving pipeline is complete.

One thing I wanted REXA to eventually have was long-term memory . The idea is simple: when I explicitly tell REXA something like: “Remember that I prefer PostgreSQL for my backend projects.” REXA should be able to save that information so it can be used later. The important distinction is that REXA does not currently retrieve these memories yet . Today, the implemented part is the memory-saving pipeline . Retrieval is the next part I plan to build. I also didn't put the memory database directly inside the REXA CLI. Instead, I built the memory infrastructure behind the REXA website backend and let the CLI communicate with it through an authenticated HTTP API. The architecture looks like this: REXA CLI │ │ POST + Bearer Token │ { "text": "..." } ▼ REXA Website Backend │ ├── Verify Token ├── Identify User ├── Process Text ├── Generate Embeddings └── Store Memory │ ▼ PostgreSQL + pgvector One important design decision is that REXA does not automatically save every preference or every piece of conversation. Instead, the model uses a tool called: save memory When the user explicitly asks REXA to remember something, the model calls this tool. For example: User: Remember that I use Bun for my backend projects. REXA can decide that the appropriate action is to call: save memory ... So memory creation is currently explicit , rather than REXA silently storing everything the user says. save memory The save memory tool receives the text that the user wants to store. Before sending it to the backend, the CLI performs some basic validation. The CLI: This means extremely large documents may never leave the CLI in the first place. For normal memory entries, however, the text can then be sent to the backend. The REXA CLI does not directly access PostgreSQL or pgvector. Instead, it sends a POST request to the REXA backend: POST https://rexa-server.onrender.com/api/cli/memory Authorization: Bearer <token Content-Type: application/json The request body contains only the memory text: { "text": "I prefer PostgreSQL for my backend projects." } Notice that there is no userId in the request body . The identity of the user comes from the authentication token. The token comes from the REXA CLI login flow. The CLI first authenticates through the REXA backend using: POST /api/cli/verify The same Bearer-token mechanism is then used when making authenticated CLI requests. This means the CLI doesn't simply tell the backend: { "userId": "123", "text": "..." } Instead, it says, effectively: Here is my authentication token. Here is the memory I want to save. The backend is responsible for determining who that token belongs to. When the memory request reaches: /api/cli/memory the backend first checks the Bearer token. If the token is missing, invalid, or expired, the request is rejected. The CLI can surface these authentication failures and provide a hint to run: rexa login This makes the authentication layer separate from the actual memory-processing logic. After successful token verification, the backend knows which authenticated user is making the request. This is important because memory is per user . The CLI doesn't provide the user identity manually. Instead: Bearer Token ↓ Token Verification ↓ Authenticated User ↓ Memory belongs to that user This prevents the client from simply claiming that a memory belongs to some other user. Once authentication succeeds, the backend receives the text. "I prefer PostgreSQL for my backend projects." From here, the rest of the pipeline happens on the backend. This is where the database and embedding infrastructure come into play. The CLI doesn't need to know how the backend implements this processing. Conceptually, the pipeline is: Text ↓ Chunking ↓ Batch Creation ↓ Embedding Generation ↓ Vector ↓ PostgreSQL + pgvector These are backend responsibilities. The backend can split larger pieces of text into smaller chunks. Conceptually: Large Text │ ├── Chunk 1 ├── Chunk 2 ├── Chunk 3 └── ... For a tiny memory such as: "I prefer PostgreSQL." there may not be much to split. But chunking becomes useful as the amount of stored information grows. After chunking, the backend can group chunks into batches for embedding. Chunks │ ├── Batch 1 │ ├── Chunk 1 │ ├── Chunk 2 │ └── Chunk 3 │ └── Batch 2 ├── Chunk 4 ├── Chunk 5 └── Chunk 6 These batches are then passed to the embedding stage. The text is converted into a numerical vector using an embedding model. "I prefer PostgreSQL for my backend projects." │ ▼ Embedding Model │ ▼ 0.12, -0.38, 0.74, ... The resulting vector represents the semantic information contained in the text. This is what makes vector-based memory possible. Instead of treating a memory as only a string of characters, the backend also stores a mathematical representation of its meaning. The generated vector is stored using pgvector alongside the memory data. The database layer uses: PostgreSQL + pgvector + Prisma A simplified representation might look like: Memory ───────────────────────────── user id text embedding created at ... The exact schema can evolve, but the important part is that the memory is associated with the authenticated user and has a vector representation that can later be used for semantic retrieval. Once the memory has been successfully processed and stored, the API returns a success response. The actual response from the controller is: { "success": true, "message": "Data saved in memory" } The CLI can then use that result to tell the user that the memory was saved successfully. So the complete flow is: User │ │ "Remember this..." ▼ REXA model │ │ calls save memory ▼ REXA CLI │ │ validate + trim text │ │ POST /api/cli/memory │ Authorization: Bearer <token │ { "text": "..." } ▼ REXA Backend │ │ verify token │ identify user │ │ process memory │ ├── chunk │ ├── batch │ ├── embed │ └── store vector ▼ PostgreSQL + pgvector │ │ success ▼ REXA Backend │ │ { "success": true, │ "message": "Data saved in memory" } ▼ REXA CLI The main architectural decision was to keep REXA itself separate from the memory infrastructure . The CLI is responsible for interacting with the agent and invoking save memory . The backend handles authentication and memory processing. PostgreSQL and pgvector provide the persistent storage and vector representation. So the responsibilities are roughly separated like this: REXA CLI → Agent interaction + save memory Backend → Authentication + memory processing PostgreSQL + pgvector → Persistent memory storage This also means the CLI doesn't need database credentials or direct database access. It only needs an authenticated API connection. At the moment, the implemented functionality is the save side of memory. User ↓ save memory ↓ Authenticated API ↓ Embedding ↓ Vector storage The retrieval side is not implemented in the CLI yet. The next step is to build the other half: Current: SAVE User ↓ save memory ↓ Backend ↓ Vector Database Future: RECALL Current Task ↓ Memory Search ↓ Similarity / Relevance ↓ Relevant Memories ↓ REXA That is where things become much more interesting. The goal isn't simply to give REXA a database full of memories. The real goal is to build a system where REXA can eventually find the right memory when it is actually useful . And that's the part I'm building next.