Four Ways Your Background Job Disappears (And How to Stop Each One) Maneshwar, developer of LiveReview, a blast-radius aware AI code review tool, explains common failure modes in background job processing, focusing on the dual write problem where a process crash between database and queue writes can silently lose jobs. The post outlines solutions such as decoupling email sending from signup requests and using queues, while highlighting the need for transactional outbox patterns to ensure reliability. Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. A user signs up on your app. They type an email and a password, they hit submit, and they expect to be inside the product roughly immediately. Somewhere in that flow, a welcome email has to go out. That single sentence, "and also send a welcome email", is one of the great load-bearing lies of backend engineering. It sounds like a footnote. It is actually an entire subsystem, and if you build it the obvious way, it will teach you that the hard way. So let's build it the obvious way first, and then keep breaking it until it stops breaking. The naive version is beautiful in its simplicity. One function, top to bottom. python @app.post "/signup" def signup payload : user = db.insert user payload 8ms email.send welcome user.email 40ms? 2s? forever? return {"ok": True, "user id": user.id} finally Read that middle line again, because it is where your uptime goes to die. On your laptop this is flawless. Your laptop has never met a rate limit. In production, that email.send welcome call is a network round trip to a company you do not control, on a bad day for them. Three things follow: It is slow. Your signup is now as slow as the mail provider's worst percentile. You have handed your p99 to somebody else's on-call rotation. It fails. When the provider 500s, your handler raises, and the whole request fails. The user sees "something went wrong" for an account that, depending on where your transaction boundary sits, may or may not actually exist now. It couples your availability to theirs. Every additional dependency in a serial request path multiplies your failure probability https://landing.google.com/sre/sre-book/chapters/embracing-risk/ . Two services at 99.9% chained together are 99.8%. You are not "using" a mail provider, you are inheriting it. The real problem here is conceptual, not technical. Creating an account and sending an email are two different pieces of work with two completely different urgency profiles. The user is waiting for the first one. Nobody, in the history of software, has ever sat and waited for a welcome email. You just glued them together anyway. The fix is to stop doing the second thing during the request. Save the user, respond immediately, and drop a note somewhere saying "an email needs sending". A separate pool of workers reads those notes and does the actual sending, at whatever pace the mail provider allows. That note-holder is a queue: SQS https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html , RabbitMQ, Redis with a proper library on top, whatever you like. Response time drops from "however long the mail API feels like today" to about 40ms. If the mail provider is down for an hour, jobs pile up in the queue and drain when it comes back. Nobody signing up even notices. This is a genuinely huge win. It is also where most tutorials stop, and it is where the interesting bug lives. Look at the bottom half of that diagram. Your handler now performs two writes to two different systems. It inserts the user into Postgres, and it publishes a message to SQS. There is no transaction spanning both, because there cannot be. They are different databases owned by different vendors that have never heard of each other. So what happens if the process dies in between? The account exists. The job does not. That user is now permanently welcome-email-less, and there is no error anywhere, no alert, no failed request. From every system's point of view, everything went fine. This is the dual write problem https://www.confluent.io/blog/transactional-outbox-pattern-real-time-data-processing/ , and it is nastier than a crash because it is silent. Reverse the order and you get the mirror image: a job to email a user who does not exist. The fix has a name that sounds far more intimidating than it is. The transactional outbox https://microservices.io/patterns/data/transactional-outbox.html . Stop writing to the queue from your handler. Write to a table instead. The same database, in the same transaction as the user row. BEGIN; INSERT INTO users id, email VALUES 'u 881', 'ada@example.com' ; INSERT INTO outbox id, type, payload VALUES 'j 204', 'send welcome', '{"user id":"u 881"}' ; COMMIT; Now there is exactly one write, to exactly one system, guarded by exactly one commit. Either the user and the job both exist, or neither does. The crack is gone because there is no longer a gap to crash into. A separate relay process then reads unsent outbox rows and publishes them to the real queue, either by polling the table or by tailing the write ahead log with something like Debezium https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html . Here is the part people gloss over, and I want to be honest about it. The outbox does not give you exactly-once. The relay can publish a message and then crash before it marks the row as sent. On restart, it publishes the same job again. You have moved the failure from "silently lose the job" to "occasionally do the job twice", which is a spectacular trade, because one of those is fixable and the other one is invisible. Hold that thought, it comes back in ten paragraphs. New failure, further down the pipe. A worker pulls a job off the queue and starts sending. Halfway through, the pod gets evicted, or the deploy rolls, or the spot instance is reclaimed. The worker is gone. Where is the job? If your queue deletes messages the moment they are handed out, the answer is nowhere. It was consumed. It is not in the queue, it is not done, and nobody is going to look for it. Real queues do not work that way. They use a visibility timeout https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html . When a worker receives a job, the job is hidden, not deleted. It sits invisible for some window, say 30 seconds. Two things can happen: Nothing is lost. Ever. The delete is the receipt , not the checkout. One practical note: set the timeout longer than your slowest realistic job, or you get the fun scenario where a job that takes 45 seconds is redelivered at second 30 and now two workers are doing it in parallel, both convinced they are alone. Look at what you have just guaranteed. A job is never lost. Notice that this is a strictly weaker promise than "a job runs exactly once", and the queue is not even pretending otherwise. Worker sends the email. Worker crashes in the microsecond before calling delete. Timer expires. Second worker sends the email again. Every distributed queue worth using is at-least-once, because exactly-once delivery across a network is not a thing you can buy https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ . You can only build exactly-once effects on top of at-least-once delivery, and that is done in the worker, not in the queue. Which means your workers have to be idempotent. Give every job a stable id at creation time, in the outbox row. The worker records that id when the work completes, and checks it before starting: python def handle job : the unique index does the arguing for us inserted = db.execute "INSERT INTO processed jobs job id VALUES %s ON CONFLICT DO NOTHING", job.id, if inserted.rowcount == 0: return someone already did this one. go home. email.send welcome job.payload "user id" A unique constraint is doing the real work here, which is the correct amount of cleverness for this problem. If two workers race, the database picks a winner. That is what it is for. Two things people get wrong: Where you write the marker matters. If the marker is in a different store than the side effect, you just recreated the dual write problem one layer down. Turtles. Not every job needs this machinery. SET last login = now is naturally idempotent. Running it twice changes nothing. INCREMENT credits BY 10 very much is not. Know which of your jobs are which, because idempotency you do not need is just latency. Some jobs are not unlucky. They are doomed. The email address is bob@@gmial.con . The account was deleted. The payload references a row that no longer exists. You can retry that job every 30 seconds until the heat death of the universe and it will fail every single time, cheerfully, forever, while burning quota and filling your logs. So you need two different behaviours for two different kinds of failure. Transient failures get retried with exponential backoff plus jitter. A 429, a timeout, a 503. The provider is having a moment and will be fine shortly. Back off so you are not part of the reason it is having a moment, and jitter so that ten thousand queued jobs do not all retry on the same second and knock it back over. AWS wrote a canonical piece on this https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ . Permanent failures should not be retried at all. A malformed address is not going to become well formed on attempt four. And after some bounded number of attempts, usually about five, the job stops. It does not get retried forever, and it absolutely does not get silently dropped. It goes to a dead letter queue https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html . The DLQ is a parking lot, not a graveyard. Jobs sit there with their payload and their failure history intact so a human can look at them, work out what went wrong, fix the cause, and replay them. I will say the important part loudly, because I have watched teams get this wrong: a dead letter queue that nobody is paged for is just a slower, more expensive way of losing data. Alarm on DLQ depth greater than zero. If nothing ever pages you, either your system is perfect or your alarm is broken, and I know which way I would bet. Here is the whole lifecycle in one picture: php stateDiagram-v2 -- Pending: written to outbox in the txn Pending -- Queued: relay publishes Queued -- InFlight: worker receives, hidden 30s InFlight -- Done: sent, then deleted InFlight -- Queued: worker crashed, timer expired InFlight -- Retrying: transient failure Retrying -- Queued: after backoff Retrying -- DLQ: 5 attempts used up InFlight -- DLQ: permanent failure, fail fast DLQ -- Pending: a human replays it Done -- Notice that InFlight has three exits and only one of them is success. That ratio is the entire job of a background system. Four versions later, here is the thing you actually ship: And the decision path when you are staring at a piece of work wondering where it belongs: php flowchart TD A a piece of work -- B{is the user waiting