{"slug": "module-of-the-week-persistedqueue", "title": "Module of the Week - PersistedQueue", "summary": "Effect's PersistedQueue module was named Module of the Week for its ability to persist background jobs across API and worker restarts, using a lock acquired via `take` that other workers cannot claim until it is released or expires. The pick followed a conversation with Adam Rankin of Warp about `effect-mq`, an Effect-native alternative to BullMQ for processing background jobs, during which Rankin said he had not heard of PersistedQueue but would likely have used it. The module decouples the queue from services so API replicas and workers can scale independently, with crashed workers' locks expiring so another worker can retry the job.", "body_md": "This week’s pick came out of a conversation with Adam Rankin from Warp on our last Effect Office Hours. We spoke about [`effect-mq`](https://www.effect-mq.com/), an Effect-native alternative to BullMQ for processing background jobs. My first thought was that it was probably using Effect’s `PersistedQueue` under the hood.\n\nTurns out, Adam hadn’t heard of the module, but wished he’d known about it sooner because he probably *would* have used it. Who wouldn’t - amirite?\n\nAfter that conversation, `PersistedQueue` was a no-brainer for our first Module of the Week.\n\n## \n\nImagine we’re building a service that removes the AI-generated nonsense from LinkedIn posts until something human remains. Users submit their posts, and we strip out the “humbled and honored,” the “let that sink in,” and whatever a missed flight apparently taught someone about servant leadership.\n\nWe can let the user know their response was successful and excavate an actual thought later.\n\nWhen a post is submitted, we:\n\n1. Enqueue a de-slop job in `PersistedQueue` .\n2. Respond to the user confirming their submission was successful.\n3. Process the post in the background when a worker picks up the job.\n\nA worker processes the text in the background and saves the result. Ideally, there’s still a sentence left.\n\nBecause the job is persisted, it survives API and worker restarts. If a worker crashes halfway through “leveraging authentic human connection,” another can retry the job. The user doesn’t have to submit it again. The next worker, unfortunately, has to re-read the whole thing.\n\nHere’s that flow with one API and one worker:\n\n## \n\nNow imagine someone shares our service on LinkedIn with an 800-word post about the importance of brevity. Submissions start pouring in, and our lone worker can’t keep up.\n\nWe need more API replicas to handle incoming submissions and horizontal autoscaling for our workers, bringing them online as the backlog grows and scaling them back down once they contain all the thought leadership.\n\nBecause the `PersistedQueue` is decoupled from these services, they can be scaled independently without affecting the queue.\n\n### \n\nHere’s our service with three API replicas feeding jobs into the same queue. As the backlog grows, our infrastructure brings two more workers online to help.\n\n### \n\nWith several workers running, what’s to stop two of them from claiming the same post? Nobody needs two processes investigating what a cold shower taught a founder about product-market fit.\n\nWhen a worker claims a job through `take`, it acquires a lock. Other workers can’t claim that job until the lock is released or expires.\n\nThe worker periodically refreshes the lock, extending its expiry so longer jobs have time to finish. Some posts have a twelve-paragraph origin story before they get to the webinar link. These things take time.\n\nIf the worker crashes, the refreshes stop and the lock eventually expires, allowing another worker to claim the job and try again.\n\nWatch the lock countdown reset when the lock is refreshed. Once worker 1 goes offline, the lock expires and worker 2 claims the job on a later poll.\n\n## \n\nLet’s put the de-slopper to work. We’ll run the submission API and worker as separate processes, with a shared queue between them.\n\n### \n\nBoth processes will use a `PostsQueue` service backed by the same named queue, `\"posts\"`.\n\nEach job carries the submitted text and a post ID to save the result against. The schema requires non-empty text, which shouldn’t be a problem. Nobody on LinkedIn has ever used zero words when six paragraphs would do.\n\nWe’ll use `QueueLayer` in both processes to connect them to the same Redis server. The API and worker don’t need to run on the same machine, as long as they can both reach that server. Replace `redis://redis:6379` with your Redis address.\n\nEffect’s dependency injection lets us swap the storage layers without changing the API or worker code. `QueueLayerTest` uses an in-memory store for tests; we could also replace the Redis layers with a SQL store and its database connection layer if we wanted.\n\n### \n\nThe API `yield*` s the `PostsQueue` to get access to the service, and then calls `offer` to enqueue the submitted post.\n\nWe also provide a stable job `id` so that we can prevent duplicates if the submission is retried. Being “beyond thrilled” once was plenty.\n\nOnce `offer` succeeds, the API can respond to the user confirming the submission. The worker will deal with all the prepositions asynchronously.\n\n### \n\nThe worker gets the same `PostsQueue` service and calls `take`, delegating the actual cleanup of a post to a hypothetical `Deslopper` service. Its `process` method removes the filler and saves the result under the job’s `postId`, replacing any previous result if the job runs again. For this post, we’re hoping for “I got promoted.”\n\nIf there’s no work available, `take` waits. Otherwise, it passes the job to `deslopper.process(job)` and acknowledges it when processing succeeds.\n\nIf processing fails with a `DeslopError`, we log it and keep the worker running. The queue’s retry settings determine whether and when the failed job becomes available again.\n\n## \n\nSuppose the text-processing service is temporarily unavailable and our handler fails. We’d like to try again, but leave some time between requests. Somewhere, a post still contains “I’m delighted to share.” It can remain delighted for another five seconds.\n\n`PersistedQueue.make` gives us two options for this: `maxAttempts` and `retrySchedule`.\n\n### \n\nLet’s allow three attempts in total, waiting five seconds before each retry:\n\n`maxAttempts: 3` allows the initial attempt and up to two retries. If the third attempt fails, the queue marks the job as failed and stops offering it to workers. We gave “unlocking human potential” three chances to become a sentence. That’s enough compute for today.\n\n### \n\nWorker 1 takes post `#041`, while worker 2 handles the other posts in the queue. Then worker 1’s handler fails. Let’s see how `#041` gets another attempt:\n\nOnce the failure reaches the queue, it releases the lock and schedules a retry in five seconds. Worker 2 has finished the other posts, but can’t claim `#041` until that delay ends.\n\nWorker 2 then claims it for attempt 2, processes it, and acknowledges completion. Two attempts, one post, three sentences about “authentic leadership” that nobody has to read again.\n\n## \n\nLock behavior is configurable depending on the store implementation that is being used.\n\nFor example, let’s say we want to refresh every 10 seconds and expire the lock after 30 seconds without a successful refresh. With the Redis store, that would look like:\n\nKeep the refresh interval shorter than the expiration, with some room for network delays or other overhead. As we’ve seen, each successful refresh will reset the expiration countdown.\n\nA shorter expiration lets another worker pick up abandoned jobs sooner. But it also means a transient connectivity issue could result in a worker losing its lock while it’s still processing a job.\n\n## \n\nTo summarize, a `PersistedQueue` is a place where you can store work and process it later. If your app makes use of background job processing, it’s definitely worth a look.\n\nWe hope you enjoyed this edition of Module of the Week! Until the next one - Happy Effecting!", "url": "https://wpnews.pro/news/module-of-the-week-persistedqueue", "canonical_source": "https://effect.website/blog/module-of-the-week/persisted-queue/", "published_at": "2026-09-15 00:00:00+00:00", "updated_at": "2026-09-16 00:07:28.968109+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["PersistedQueue", "Effect", "Adam Rankin", "Warp", "effect-mq", "BullMQ", "Redis", "QueueLayer"], "alternates": {"html": "https://wpnews.pro/news/module-of-the-week-persistedqueue", "markdown": "https://wpnews.pro/news/module-of-the-week-persistedqueue.md", "text": "https://wpnews.pro/news/module-of-the-week-persistedqueue.txt", "jsonld": "https://wpnews.pro/news/module-of-the-week-persistedqueue.jsonld"}}