Why the Outbox Pattern, Queue, and Embedding Worker?

A developer building the Second-Memory semantic search system decoupled memory creation from embedding generation using a transactional outbox pattern, a queue, and a dedicated embedding worker writing to pgvector. Memory records and outbox events are written in a single PostgreSQL transaction, so embeddings can be retried and scaled independently without blocking the memory API, at the cost of eventual consistency between a memory and its vector. The developer accepted that trade-off, noting immediate persistence matters more than making a new memory instantly searchable.

The previous post covered why I chose pgvector for semantic search. But there was another question: When should a memory be embedded? At first, it might seem simple: Create memory ▼ Generate embedding ▼ Save embedding But this makes memory creation dependent on the embedding process. If the embedding provider is slow or unavailable, creating a memory could also fail or become slow. I wanted to separate these two operations. The architecture became: Memory Service ▼ PostgreSQL ▼ Outbox Table ▼ Queue / Job Runner ▼ Embedding Worker ▼ pgvector When a user creates a memory, the Memory Service writes the memory and an outbox event in the same database transaction. For example: Transaction ┌─────────────────────────────┐ │ INSERT memory │ │ INSERT outbox event │ └─────────────────────────────┘ The important part is that they succeed or fail together. This avoids a common problem with distributed systems: Memory saved ▼ Embedding event lost The outbox gives me a durable record of the work that needs to happen. The outbox is not the queue itself. It is a reliable bridge between the database transaction and asynchronous processing. A separate process reads pending outbox events and puts jobs onto a queue. Outbox ▼ Queue ▼ Embedding Worker This gives the embedding process some useful properties: asynchronous processing retries independent scaling failure isolation no need to block the memory API The user can save a memory without waiting for the embedding provider. The embedding worker has one main responsibility: turn memory text into an embedding and store it in pgvector. php flowchart TD A Get Memory -- B Generate embedding B -- C Store Vector This keeps embedding-specific logic out of the Memory Service's synchronous request path. It also gives me a place to evolve the embedding pipeline later. For example, I could change: embedding models chunking strategy retry behaviour batch processing embedding dimensions without changing the API used to create a memory. There is an important consequence of this design: A newly created memory may not be immediately searchable. There can be a small delay between: Memory created │ asynchronous processing ▼ Embedding created ▼ Memory available for semantic search I accepted this trade-off. For Second-Memory, immediate persistence is more important than making the embedding operation part of the user's request. This is essentially eventual consistency between the memory record and its vector representation. The asynchronous design also changes how failures work. If the embedding provider temporarily fails: php Memory ▼ Outbox ▼ Queue ▼ Embedding Worker - X - Retry The memory itself has already been safely stored. The embedding job can be retried without asking the user to submit the memory again. That separation was important to me. The final flow became: php graph TD CM Create Memory -- PG subgraph PG PostgreSQL M Memory OE Outbox Event end OE -- Q Queue Q -- EW Embedding Worker EW -- PV pgvector in PostgreSQL Each part has a different responsibility: | Component | Responsibility | |---|---| | Memory Service | Store the memory | | Outbox | Reliably record the event | | Queue | Deliver asynchronous work | | Embedding Worker | Generate embeddings | | pgvector | Store and search vectors | This added some complexity compared with simply generating the embedding inside the API request. But it gave me something more important: memory creation is no longer tightly coupled to the availability of the embedding pipeline. That was the trade-off I wanted.