{"slug": "why-the-outbox-pattern-queue-and-embedding-worker", "title": "Why the Outbox Pattern, Queue, and Embedding Worker?", "summary": "A developer building the Second-Memory semantic search system decoupled memory creation from embedding generation using a transactional outbox pattern, a queue, and a dedicated embedding worker writing to pgvector. Memory records and outbox events are written in a single PostgreSQL transaction, so embeddings can be retried and scaled independently without blocking the memory API, at the cost of eventual consistency between a memory and its vector. The developer accepted that trade-off, noting immediate persistence matters more than making a new memory instantly searchable.", "body_md": "The previous post covered why I chose pgvector for semantic search.\n\nBut there was another question:\n\n**When should a memory be embedded?**\n\nAt first, it might seem simple:\n\n```\nCreate memory\n     ▼\nGenerate embedding\n     ▼\nSave embedding\n```\n\nBut this makes memory creation dependent on the embedding process.\n\nIf the embedding provider is slow or unavailable, creating a memory could also fail or become slow.\n\nI wanted to separate these two operations.\n\nThe architecture became:\n\n```\nMemory Service\n      ▼\n PostgreSQL\n      ▼\n Outbox Table\n      ▼\n Queue / Job Runner\n      ▼\n Embedding Worker\n      ▼\n pgvector\n```\n\nWhen a user creates a memory, the Memory Service writes the memory and an outbox event in the same database transaction.\n\nFor example:\n\n```\nTransaction\n┌─────────────────────────────┐\n│ INSERT memory               │\n│ INSERT outbox event         │\n└─────────────────────────────┘\n```\n\nThe important part is that they succeed or fail together.\n\nThis avoids a common problem with distributed systems:\n\n```\nMemory saved\n     ▼\nEmbedding event lost\n```\n\nThe outbox gives me a durable record of the work that needs to happen.\n\nThe outbox is not the queue itself.\n\nIt is a reliable bridge between the database transaction and asynchronous processing.\n\nA separate process reads pending outbox events and puts jobs onto a queue.\n\n```\nOutbox\n   ▼\nQueue\n   ▼\nEmbedding Worker\n```\n\nThis gives the embedding process some useful properties:\n\nasynchronous processing\n\nretries\n\nindependent scaling\n\nfailure isolation\n\nno need to block the memory API\n\nThe user can save a memory without waiting for the embedding provider.\n\nThe embedding worker has one main responsibility:\n\n**turn memory text into an embedding and store it in pgvector.**\n\n``` php\nflowchart TD\n    A[Get Memory] --> B[Generate embedding]\n    B --> C[Store Vector]\n```\n\nThis keeps embedding-specific logic out of the Memory Service's synchronous request path.\n\nIt also gives me a place to evolve the embedding pipeline later.\n\nFor example, I could change:\n\nembedding models\n\nchunking strategy\n\nretry behaviour\n\nbatch processing\n\nembedding dimensions\n\nwithout changing the API used to create a memory.\n\nThere is an important consequence of this design:\n\n**A newly created memory may not be immediately searchable.**\n\nThere can be a small delay between:\n\n```\nMemory created\n     │ asynchronous processing\n     ▼\nEmbedding created\n     ▼\nMemory available for semantic search\n```\n\nI accepted this trade-off.\n\nFor Second-Memory, immediate persistence is more important than making the embedding operation part of the user's request.\n\nThis is essentially **eventual consistency** between the memory record and its vector representation.\n\nThe asynchronous design also changes how failures work.\n\nIf the embedding provider temporarily fails:\n\n``` php\nMemory\n  ▼\nOutbox\n  ▼\nQueue\n  ▼\nEmbedding Worker - X -> Retry\n```\n\nThe memory itself has already been safely stored.\n\nThe embedding job can be retried without asking the user to submit the memory again.\n\nThat separation was important to me.\n\nThe final flow became:\n\n``` php\ngraph TD\n    CM[Create Memory] --> PG\n\n    subgraph PG [PostgreSQL]\n            M[Memory]\n            OE[Outbox Event]\n    end\n\n    OE --> Q[Queue]\n    Q --> EW[Embedding Worker]\n    EW --> PV[(pgvector in PostgreSQL)]\n```\n\nEach part has a different responsibility:\n\n| Component | Responsibility | \n|---|---|\n| Memory Service | Store the memory | \n| Outbox | Reliably record the event | \n| Queue | Deliver asynchronous work | \n| Embedding Worker | Generate embeddings | \n| pgvector | Store and search vectors | \n\nThis added some complexity compared with simply generating the embedding inside the API request.\n\nBut it gave me something more important:\n\n**memory creation is no longer tightly coupled to the availability of the embedding pipeline.**\n\nThat was the trade-off I wanted.", "url": "https://wpnews.pro/news/why-the-outbox-pattern-queue-and-embedding-worker", "canonical_source": "https://dev.to/joungpark/why-the-outbox-pattern-queue-and-embedding-worker-2pb5", "published_at": "2026-10-07 03:47:01+00:00", "updated_at": "2026-10-07 03:47:26.381117+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-agents", "developer-tools"], "entities": ["Second-Memory", "PostgreSQL", "pgvector"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-the-outbox-pattern-queue-and-embedding-worker", "markdown": "https://wpnews.pro/news/why-the-outbox-pattern-queue-and-embedding-worker.md", "text": "https://wpnews.pro/news/why-the-outbox-pattern-queue-and-embedding-worker.txt", "jsonld": "https://wpnews.pro/news/why-the-outbox-pattern-queue-and-embedding-worker.jsonld"}}