Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
A user signs up on your app.
They type an email and a password, they hit submit, and they expect to be inside the product roughly immediately.
Somewhere in that flow, a welcome email has to go out.
That single sentence, "and also send a welcome email", is one of the great load-bearing lies of backend engineering.
It sounds like a footnote.
It is actually an entire subsystem, and if you build it the obvious way, it will teach you that the hard way.
So let's build it the obvious way first, and then keep breaking it until it stops breaking.
The naive version is beautiful in its simplicity. One function, top to bottom.
@app.post("/signup")
def signup(payload):
user = db.insert_user(payload) # 8ms
email.send_welcome(user.email) # 40ms? 2s? forever?
return {"ok": True, "user_id": user.id} # finally
Read that middle line again, because it is where your uptime goes to die.
On your laptop this is flawless. Your laptop has never met a rate limit.
In production, that email.send_welcome
call is a network round trip to a company you do not control, on a bad day for them.
Three things follow:
It is slow. Your signup is now as slow as the mail provider's worst percentile.
You have handed your p99 to somebody else's on-call rotation.
It fails. When the provider 500s, your handler raises, and the whole request fails.
The user sees "something went wrong" for an account that, depending on where your transaction boundary sits, may or may not actually exist now.
It couples your availability to theirs. Every additional dependency in a serial request path multiplies your failure probability.
Two services at 99.9% chained together are 99.8%. You are not "using" a mail provider, you are inheriting it.
The real problem here is conceptual, not technical.
Creating an account and sending an email are two different pieces of work with two completely different urgency profiles.
The user is waiting for the first one. Nobody, in the history of software, has ever sat and waited for a welcome email.
You just glued them together anyway.
The fix is to stop doing the second thing during the request.
Save the user, respond immediately, and drop a note somewhere saying "an email needs sending".
A separate pool of workers reads those notes and does the actual sending, at whatever pace the mail provider allows.
That note-holder is a queue: SQS, RabbitMQ, Redis with a proper library on top, whatever you like.
Response time drops from "however long the mail API feels like today" to about 40ms.
If the mail provider is down for an hour, jobs pile up in the queue and drain when it comes back. Nobody signing up even notices.
This is a genuinely huge win.
It is also where most tutorials stop, and it is where the interesting bug lives.
Look at the bottom half of that diagram.
Your handler now performs two writes to two different systems.
It inserts the user into Postgres, and it publishes a message to SQS.
There is no transaction spanning both, because there cannot be.
They are different databases owned by different vendors that have never heard of each other.
So what happens if the process dies in between?
The account exists. The job does not.
That user is now permanently welcome-email-less, and there is no error anywhere, no alert, no failed request.
From every system's point of view, everything went fine.
This is the dual write problem, and it is nastier than a crash because it is silent.
Reverse the order and you get the mirror image: a job to email a user who does not exist.
The fix has a name that sounds far more intimidating than it is. The transactional outbox.
Stop writing to the queue from your handler. Write to a table instead. The same database, in the same transaction as the user row.
BEGIN;
INSERT INTO users (id, email) VALUES ('u_881', 'ada@example.com');
INSERT INTO outbox (id, type, payload) VALUES ('j_204', 'send_welcome', '{"user_id":"u_881"}');
COMMIT;
Now there is exactly one write, to exactly one system, guarded by exactly one commit. Either the user and the job both exist, or neither does. The crack is gone because there is no longer a gap to crash into.
A separate relay process then reads unsent outbox rows and publishes them to the real queue, either by polling the table or by tailing the write ahead log with something like Debezium.
Here is the part people gloss over, and I want to be honest about it.
The outbox does not give you exactly-once. The relay can publish a message and then crash before it marks the row as sent.
On restart, it publishes the same job again.
You have moved the failure from "silently lose the job" to "occasionally do the job twice", which is a spectacular trade, because one of those is fixable and the other one is invisible.
Hold that thought, it comes back in ten paragraphs.
New failure, further down the pipe.
A worker pulls a job off the queue and starts sending.
Halfway through, the pod gets evicted, or the deploy rolls, or the spot instance is reclaimed. The worker is gone.
Where is the job?
If your queue deletes messages the moment they are handed out, the answer is nowhere. It was consumed.
It is not in the queue, it is not done, and nobody is going to look for it.
Real queues do not work that way.
They use a visibility timeout.
When a worker receives a job, the job is hidden, not deleted. It sits invisible for some window, say 30 seconds. Two things can happen:
Nothing is lost. Ever. The delete is the receipt, not the checkout.
One practical note: set the timeout longer than your slowest realistic job, or you get the fun scenario where a job that takes 45 seconds is redelivered at second 30 and now two workers are doing it in parallel, both convinced they are alone.
Look at what you have just guaranteed. A job is never lost.
Notice that this is a strictly weaker promise than "a job runs exactly once", and the queue is not even pretending otherwise.
Worker sends the email. Worker crashes in the microsecond before calling delete. Timer expires. Second worker sends the email again.
Every distributed queue worth using is at-least-once, because exactly-once delivery across a network is not a thing you can buy.
You can only build exactly-once effects on top of at-least-once delivery, and that is done in the worker, not in the queue.
Which means your workers have to be idempotent.
Give every job a stable id at creation time, in the outbox row.
The worker records that id when the work completes, and checks it before starting:
def handle(job):
inserted = db.execute(
"INSERT INTO processed_jobs (job_id) VALUES (%s) ON CONFLICT DO NOTHING",
job.id,
)
if inserted.rowcount == 0:
return # someone already did this one. go home.
email.send_welcome(job.payload["user_id"])
A unique constraint is doing the real work here, which is the correct amount of cleverness for this problem.
If two workers race, the database picks a winner. That is what it is for.
Two things people get wrong:
Where you write the marker matters. If the marker is in a different store than the side effect, you just recreated the dual write problem one layer down. Turtles.
Not every job needs this machinery. SET last_login = now()
is naturally idempotent. Running it twice changes nothing.
INCREMENT credits BY 10
very much is not. Know which of your jobs are which, because idempotency you do not need is just latency.
Some jobs are not unlucky. They are doomed.
The email address is bob@@gmial.con
. The account was deleted. The payload references a row that no longer exists.
You can retry that job every 30 seconds until the heat death of the universe and it will fail every single time, cheerfully, forever, while burning quota and filling your logs.
So you need two different behaviours for two different kinds of failure.
Transient failures get retried with exponential backoff plus jitter. A 429, a timeout, a 503.
The provider is having a moment and will be fine shortly.
Back off so you are not part of the reason it is having a moment, and jitter so that ten thousand queued jobs do not all retry on the same second and knock it back over.
AWS wrote a canonical piece on this.
Permanent failures should not be retried at all. A malformed address is not going to become well formed on attempt four.
And after some bounded number of attempts, usually about five, the job stops.
It does not get retried forever, and it absolutely does not get silently dropped. It goes to a dead letter queue.
The DLQ is a parking lot, not a graveyard. Jobs sit there with their payload and their failure history intact so a human can look at them, work out what went wrong, fix the cause, and replay them.
I will say the important part loudly, because I have watched teams get this wrong: a dead letter queue that nobody is paged for is just a slower, more expensive way of losing data. Alarm on DLQ depth greater than zero.
If nothing ever pages you, either your system is perfect or your alarm is broken, and I know which way I would bet.
Here is the whole lifecycle in one picture:
stateDiagram-v2
[*] --> Pending: written to outbox in the txn
Pending --> Queued: relay publishes
Queued --> InFlight: worker receives, hidden 30s
InFlight --> Done: sent, then deleted
InFlight --> Queued: worker crashed, timer expired
InFlight --> Retrying: transient failure
Retrying --> Queued: after backoff
Retrying --> DLQ: 5 attempts used up
InFlight --> DLQ: permanent failure, fail fast
DLQ --> Pending: a human replays it
Done --> [*]
Notice that InFlight
has three exits and only one of them is success.
That ratio is the entire job of a background system.
Four versions later, here is the thing you actually ship:
And the decision path when you are staring at a piece of work wondering where it belongs:
flowchart TD
A[a piece of work] --> B{is the user waiting<br/>for the result?}
B -->|yes| C[do it in the request]
B -->|no| D{does it touch<br/>something you do not own?}
D -->|no| E{is it slow<br/>or bursty?}
D -->|yes| F[background job]
E -->|no| C
E -->|yes| F
F --> G{is running it twice<br/>harmful?}
G -->|yes| H[background job<br/>+ idempotency key]
G -->|no| I[background job<br/>ship it]
classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
classDef sync fill:#6ea8ff,stroke:#3b6dcc,color:#1a1a1a
classDef async fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
class B,D,E,G decision
class A start
class C sync
class F,H,I async
Step back from the boxes for a second, because it is easy to look at that pipeline and conclude that you have built something enormous to send one email.
You have not. You have bought four specific properties, and each one is the direct answer to a version that broke:
None of this makes email delivery reliable. Email is not reliable and never has been. What it does is make your signup independent of email, which is the only part of that sentence you were ever able to control.
The general lesson outlives the example. Any time you catch yourself writing "and also" in a request handler, and also send the email, and also update the search index, and also ping the CRM, you are describing a background job. The "also" is the tell.
Push it out of the request, write it down transactionally, make it repeatable, and give it somewhere to fail loudly.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production code safe without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase: