{"slug": "four-ways-your-background-job-disappears-and-how-to-stop-each-one", "title": "Four Ways Your Background Job Disappears (And How to Stop Each One)", "summary": "Maneshwar, developer of LiveReview, a blast-radius aware AI code review tool, explains common failure modes in background job processing, focusing on the dual write problem where a process crash between database and queue writes can silently lose jobs. The post outlines solutions such as decoupling email sending from signup requests and using queues, while highlighting the need for transactional outbox patterns to ensure reliability.", "body_md": "*Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nA user signs up on your app.\n\nThey type an email and a password, they hit submit, and they expect to be inside the product roughly immediately.\n\nSomewhere in that flow, a welcome email has to go out.\n\nThat single sentence, \"and also send a welcome email\", is one of the great load-bearing lies of backend engineering.\n\nIt sounds like a footnote.\n\nIt is actually an entire subsystem, and if you build it the obvious way, it will teach you that the hard way.\n\nSo let's build it the obvious way first, and then keep breaking it until it stops breaking.\n\nThe naive version is beautiful in its simplicity. One function, top to bottom.\n\n``` python\n@app.post(\"/signup\")\ndef signup(payload):\n    user = db.insert_user(payload)          # 8ms\n    email.send_welcome(user.email)          # 40ms? 2s? forever?\n    return {\"ok\": True, \"user_id\": user.id} # finally\n```\n\nRead that middle line again, because it is where your uptime goes to die.\n\nOn your laptop this is flawless. Your laptop has never met a rate limit.\n\nIn production, that `email.send_welcome`\n\ncall is a network round trip to a company you do not control, on a bad day for them.\n\nThree things follow:\n\n**It is slow.** Your signup is now as slow as the mail provider's worst percentile.\n\nYou have handed your p99 to somebody else's on-call rotation.\n\n**It fails.** When the provider 500s, your handler raises, and the whole request fails.\n\nThe user sees \"something went wrong\" for an account that, depending on where your transaction boundary sits, may or may not actually exist now.\n\n**It couples your availability to theirs.** [Every additional dependency in a serial request path multiplies your failure probability](https://landing.google.com/sre/sre-book/chapters/embracing-risk/).\n\nTwo services at 99.9% chained together are 99.8%. You are not \"using\" a mail provider, you are *inheriting* it.\n\nThe real problem here is conceptual, not technical.\n\nCreating an account and sending an email are two different pieces of work with two completely different urgency profiles.\n\nThe user is waiting for the first one. Nobody, in the history of software, has ever sat and waited for a welcome email.\n\nYou just glued them together anyway.\n\nThe fix is to stop doing the second thing during the request.\n\nSave the user, respond immediately, and drop a note somewhere saying \"an email needs sending\".\n\nA separate pool of workers reads those notes and does the actual sending, at whatever pace the mail provider allows.\n\nThat note-holder is a queue: [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html), RabbitMQ, Redis with a proper library on top, whatever you like.\n\nResponse time drops from \"however long the mail API feels like today\" to about 40ms.\n\nIf the mail provider is down for an hour, jobs pile up in the queue and drain when it comes back. Nobody signing up even notices.\n\nThis is a genuinely huge win.\n\nIt is also where most tutorials stop, and it is where the interesting bug lives.\n\nLook at the bottom half of that diagram.\n\nYour handler now performs **two writes to two different systems.**\n\nIt inserts the user into Postgres, and it publishes a message to SQS.\n\nThere is no transaction spanning both, because there cannot be.\n\nThey are different databases owned by different vendors that have never heard of each other.\n\nSo what happens if the process dies in between?\n\nThe account exists. The job does not.\n\nThat user is now permanently welcome-email-less, and there is no error anywhere, no alert, no failed request.\n\nFrom every system's point of view, everything went fine.\n\nThis is the [dual write problem](https://www.confluent.io/blog/transactional-outbox-pattern-real-time-data-processing/), and it is nastier than a crash because it is silent.\n\nReverse the order and you get the mirror image: a job to email a user who does not exist.\n\nThe fix has a name that sounds far more intimidating than it is. The [transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html).\n\nStop writing to the queue from your handler. Write to a table instead. The same database, in the same transaction as the user row.\n\n```\nBEGIN;\n  INSERT INTO users (id, email)          VALUES ('u_881', 'ada@example.com');\n  INSERT INTO outbox (id, type, payload) VALUES ('j_204', 'send_welcome', '{\"user_id\":\"u_881\"}');\nCOMMIT;\n```\n\nNow there is exactly one write, to exactly one system, guarded by exactly one commit. Either the user and the job both exist, or neither does. The crack is gone because there is no longer a gap to crash into.\n\nA separate relay process then reads unsent outbox rows and publishes them to the real queue, either by polling the table or by tailing the write ahead log with something like [Debezium](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html).\n\nHere is the part people gloss over, and I want to be honest about it.\n\n**The outbox does not give you exactly-once.** The relay can publish a message and then crash before it marks the row as sent.\n\nOn restart, it publishes the same job again.\n\nYou have moved the failure from \"silently lose the job\" to \"occasionally do the job twice\", which is a spectacular trade, because one of those is fixable and the other one is invisible.\n\nHold that thought, it comes back in ten paragraphs.\n\nNew failure, further down the pipe.\n\nA worker pulls a job off the queue and starts sending.\n\nHalfway through, the pod gets evicted, or the deploy rolls, or the spot instance is reclaimed. The worker is gone.\n\nWhere is the job?\n\nIf your queue deletes messages the moment they are handed out, the answer is nowhere. It was consumed.\n\nIt is not in the queue, it is not done, and nobody is going to look for it.\n\nReal queues do not work that way.\n\nThey use a [visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html).\n\nWhen a worker receives a job, the job is **hidden, not deleted.** It sits invisible for some window, say 30 seconds. Two things can happen:\n\nNothing is lost. Ever. The delete is the *receipt*, not the checkout.\n\nOne practical note: set the timeout longer than your slowest realistic job, or you get the fun scenario where a job that takes 45 seconds is redelivered at second 30 and now two workers are doing it in parallel, both convinced they are alone.\n\nLook at what you have just guaranteed. A job is never lost.\n\nNotice that this is a strictly weaker promise than \"a job runs exactly once\", and the queue is not even pretending otherwise.\n\nWorker sends the email. Worker crashes in the microsecond before calling delete. Timer expires. Second worker sends the email again.\n\nEvery distributed queue worth using is at-least-once, because exactly-once delivery across a network is [not a thing you can buy](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/).\n\nYou can only build exactly-once *effects* on top of at-least-once delivery, and that is done in the worker, not in the queue.\n\nWhich means your workers have to be idempotent.\n\nGive every job a stable id at creation time, in the outbox row.\n\nThe worker records that id when the work completes, and checks it before starting:\n\n``` python\ndef handle(job):\n    # the unique index does the arguing for us\n    inserted = db.execute(\n        \"INSERT INTO processed_jobs (job_id) VALUES (%s) ON CONFLICT DO NOTHING\",\n        job.id,\n    )\n    if inserted.rowcount == 0:\n        return  # someone already did this one. go home.\n\n    email.send_welcome(job.payload[\"user_id\"])\n```\n\nA unique constraint is doing the real work here, which is the correct amount of cleverness for this problem.\n\nIf two workers race, the database picks a winner. That is what it is for.\n\nTwo things people get wrong:\n\n**Where you write the marker matters.** If the marker is in a different store than the side effect, you just recreated the dual write problem one layer down. Turtles.\n\n**Not every job needs this machinery.** `SET last_login = now()`\n\nis naturally idempotent. Running it twice changes nothing.\n\n`INCREMENT credits BY 10`\n\nvery much is not. Know which of your jobs are which, because idempotency you do not need is just latency.\n\nSome jobs are not unlucky. They are doomed.\n\nThe email address is `bob@@gmial.con`\n\n. The account was deleted. The payload references a row that no longer exists.\n\nYou can retry that job every 30 seconds until the heat death of the universe and it will fail every single time, cheerfully, forever, while burning quota and filling your logs.\n\nSo you need two different behaviours for two different kinds of failure.\n\n**Transient failures** get retried with exponential backoff plus jitter. A 429, a timeout, a 503.\n\nThe provider is having a moment and will be fine shortly.\n\nBack off so you are not part of the reason it is having a moment, and jitter so that ten thousand queued jobs do not all retry on the same second and knock it back over.\n\n[AWS wrote a canonical piece on this](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/).\n\n**Permanent failures** should not be retried at all. A malformed address is not going to become well formed on attempt four.\n\nAnd after some bounded number of attempts, usually about five, the job stops.\n\nIt does not get retried forever, and it absolutely does not get silently dropped. It goes to a [dead letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html).\n\nThe DLQ is a parking lot, not a graveyard. Jobs sit there with their payload and their failure history intact so a human can look at them, work out what went wrong, fix the cause, and replay them.\n\nI will say the important part loudly, because I have watched teams get this wrong: **a dead letter queue that nobody is paged for is just a slower, more expensive way of losing data.** Alarm on DLQ depth greater than zero.\n\nIf nothing ever pages you, either your system is perfect or your alarm is broken, and I know which way I would bet.\n\nHere is the whole lifecycle in one picture:\n\n``` php\nstateDiagram-v2\n    [*] --> Pending: written to outbox in the txn\n    Pending --> Queued: relay publishes\n    Queued --> InFlight: worker receives, hidden 30s\n    InFlight --> Done: sent, then deleted\n    InFlight --> Queued: worker crashed, timer expired\n    InFlight --> Retrying: transient failure\n    Retrying --> Queued: after backoff\n    Retrying --> DLQ: 5 attempts used up\n    InFlight --> DLQ: permanent failure, fail fast\n    DLQ --> Pending: a human replays it\n    Done --> [*]\n```\n\nNotice that `InFlight`\n\nhas three exits and only one of them is success.\n\nThat ratio is the entire job of a background system.\n\nFour versions later, here is the thing you actually ship:\n\nAnd the decision path when you are staring at a piece of work wondering where it belongs:\n\n``` php\nflowchart TD\n    A[a piece of work] --> B{is the user waiting<br/>for the result?}\n    B -->|yes| C[do it in the request]\n    B -->|no| D{does it touch<br/>something you do not own?}\n    D -->|no| E{is it slow<br/>or bursty?}\n    D -->|yes| F[background job]\n    E -->|no| C\n    E -->|yes| F\n    F --> G{is running it twice<br/>harmful?}\n    G -->|yes| H[background job<br/>+ idempotency key]\n    G -->|no| I[background job<br/>ship it]\n\n    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a\n    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a\n    classDef sync fill:#6ea8ff,stroke:#3b6dcc,color:#1a1a1a\n    classDef async fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a\n\n    class B,D,E,G decision\n    class A start\n    class C sync\n    class F,H,I async\n```\n\nStep back from the boxes for a second, because it is easy to look at that pipeline and conclude that you have built something enormous to send one email.\n\nYou have not. You have bought four specific properties, and each one is the direct answer to a version that broke:\n\nNone of this makes email delivery reliable. Email is not reliable and never has been. What it does is make **your signup** independent of email, which is the only part of that sentence you were ever able to control.\n\nThe general lesson outlives the example. Any time you catch yourself writing \"and also\" in a request handler, and also send the email, and also update the search index, and also ping the CRM, you are describing a background job. The \"also\" is the tell.\n\nPush it out of the request, write it down transactionally, make it repeatable, and give it somewhere to fail loudly.\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production code safe without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n**Try LiveReview on your codebase:**", "url": "https://wpnews.pro/news/four-ways-your-background-job-disappears-and-how-to-stop-each-one", "canonical_source": "https://dev.to/lovestaco/your-welcome-email-is-not-part-of-signup-designing-a-background-job-system-1lmo", "published_at": "2026-09-03 12:10:26+00:00", "updated_at": "2026-09-03 12:25:59.937897+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["LiveReview", "Maneshwar", "SQS", "RabbitMQ", "Redis", "Postgres"], "alternates": {"html": "https://wpnews.pro/news/four-ways-your-background-job-disappears-and-how-to-stop-each-one", "markdown": "https://wpnews.pro/news/four-ways-your-background-job-disappears-and-how-to-stop-each-one.md", "text": "https://wpnews.pro/news/four-ways-your-background-job-disappears-and-how-to-stop-each-one.txt", "jsonld": "https://wpnews.pro/news/four-ways-your-background-job-disappears-and-how-to-stop-each-one.jsonld"}}