{"slug": "designing-a-webhook-delivery-system-for-10-million-events-a-day", "title": "Designing a Webhook Delivery System for 10 Million Events a Day", "summary": "Maneshwar, the developer behind LiveReview, an AI code review tool, detailed the design of a webhook delivery system capable of handling 10 million events per day. The article outlines the pitfalls of naive implementations, such as synchronous HTTP calls in the request path, and advocates for the transactional outbox pattern to ensure durability and prevent head-of-line blocking.", "body_md": "*Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nSomebody on your team is going to say it, probably in a planning meeting, probably while looking at a Jira ticket that has three words in it.\n\n\"Webhooks? That's just a POST request. Half a day.\"\n\nAnd they are not wrong about the POST request part.\n\nThat is genuinely all a webhook is.\n\nSomething happened on your side, your customer wants to know, you send them an HTTP request. Done.\n\nThen you ship it, and six weeks later you are on a call explaining to a customer why they missed 4,000 payment events during a window when *their own server* was down.\n\nSo let's actually build this thing.\n\nTen million events a day, which is roughly 115 a second on average and a lot more than that at peak.\n\nI'm going to build the naive version first and then break it, on purpose, over and over, until we end up somewhere that survives contact with real customers.\n\nThe obvious one. Event happens in your request handler, you post it to the customer's URL, you wait for a `200 OK`\n\n.\n\n``` python\ndef on_payment_succeeded(payment):\n    db.save(payment)\n    requests.post(customer.webhook_url, json=payment.to_dict())  # 🙃\n    return {\"ok\": True}\n```\n\nThis works beautifully in staging, where the \"customer\" is a webhook.site tab you have open in another window.\n\nHere is what it looks like in production.\n\nYour customer's endpoint is a Rails app on a small box that also runs their cron jobs.\n\nAt 3pm their cron kicks off, their server starts taking eight seconds to respond, and now *your* request handler is sitting there holding a thread hostage waiting on somebody else's infrastructure.\n\nYour latency graph spikes. Your connection pool drains.\n\nYour own users, who have nothing to do with this, start seeing timeouts.\n\nAnd then the worst part: your request times out, the process moves on, and the event is gone.\n\nYou never wrote it down anywhere. It existed only as a variable in a function that has now returned.\n\nThe bug here is not \"it was slow.\" The bug is that you made your availability a function of your customer's availability, and you did it in the hot path.\n\nRule one of distributed systems, and honestly rule one of life: write it down before you try to do it.\n\nSo the handler stops posting. It writes the event to a table, in the same transaction as the business change that caused it, and returns.\n\nThat is the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html), and it is doing something subtle that is worth saying out loud.\n\nIf you save the payment and then push to a queue, those are two systems and there is a gap between them.\n\nCrash in the gap, and you have a payment with no event.\n\nBy putting the event row in the *same* database transaction as the payment, the two either both happen or both don't. No gap.\n\n```\nBEGIN;\n  INSERT INTO payments (id, amount, status) VALUES (...);\n  INSERT INTO webhook_outbox (customer_id, event_type, payload, status)\n       VALUES (..., 'payment.succeeded', ..., 'pending');\nCOMMIT;\n```\n\nThen a separate worker polls for `pending`\n\nrows and does the actual posting.\n\nYour handler is fast again. Your event is durable.\n\nIf the delivery fails, it fails somewhere you can see and retry, instead of in a dead stack frame.\n\nBut you have traded one problem for a sneakier one.\n\nYou have one worker, or one pool of workers, pulling from one queue in order.\n\nCustomer A's endpoint takes 10 seconds to time out.\n\nEvery worker that picks up a Customer A job is parked for 10 seconds.\n\nMeanwhile Customers B through Z have events sitting behind them in the queue, perfectly deliverable, going nowhere.\n\nThis is head-of-line blocking, and it is the noisy neighbour problem wearing a queue costume.\n\nOne customer with a bad endpoint degrades everyone. Your worst customer sets the pace for all of them.\n\nThe fix is fairness, and fairness needs somebody to enforce it.\n\nPut a dispatcher in front of the worker pool.\n\nIts whole job is to decide *which event goes next*, and it is not allowed to just take the oldest one.\n\nThe dispatcher keeps a count of how many workers are currently busy with each customer.\n\nCustomer A already has 3 in flight and their cap is 3? Skip them. Take the next customer's event instead. Come back to A later.\n\nThis is per-tenant concurrency limiting, and it is the single highest-leverage thing in the whole design.\n\nConcurrency limits are also what [Stripe uses](https://stripe.com/blog/rate-limiters) as a first-class rate limiting primitive, for exactly this reason: they bound damage rather than just counting requests.\n\nThe effect is that Customer A's disaster is now capped. Three workers are stuck on them.\n\nEvery other worker in the pool is happily serving everyone else.\n\nA slow customer now only degrades themselves, which is the correct place for the pain to land.\n\nIf you want to go further, the dispatcher is also where you put weighted fairness, so your enterprise tier doesn't get starved by a free-tier customer emitting a million events an hour.\n\nHere is the routing logic, which is really the heart of the system:\n\n``` php\nflowchart TD\n    A[Pull next pending event] --> B{Customer at<br/>concurrency cap?}\n    B -->|Yes| C[Skip, try next customer]\n    B -->|No| D{Endpoint circuit<br/>open?}\n    D -->|Yes| E[Park until cooldown ends]\n    D -->|No| F[Hand to a free worker]\n    C --> A\n    E --> A\n    F --> G[POST signed payload]\n\n    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a\n    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a\n    classDef action   fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a\n    classDef wait     fill:#ff9a5c,stroke:#c26a33,color:#1a1a1a\n\n    class B,D decision\n    class A start\n    class F,G action\n    class C,E wait\n```\n\nThat circuit breaker branch is worth adding once you have the concurrency cap working.\n\nIf a customer's endpoint has failed the last 20 attempts in a row, you already know the next one fails too.\n\nStop spending workers to find out.\n\nZoom all the way in now. A worker has picked up one job. What happens?\n\nShort timeout. Ten seconds, not sixty. A slow endpoint is a broken endpoint and you should not let it hold a worker hostage while you find out.\n\nThen you look at what comes back, and the important move is that **not all failures are the same failure.**\n\n`200`\n\n, `201`\n\n, `204`\n\n: delivered. Mark it, move on.`502`\n\n, `503`\n\n, `429`\n\n: temporary. Their server is having a moment. Retry with exponential backoff, with jitter, so that when their box comes back up you don't hit it with your entire retry backlog in the same millisecond. AWS wrote `404`\n\n, `410`\n\n, DNS does not resolve, TLS handshake fails: permanent. The URL is wrong, or the endpoint is gone. Retrying this 12 times over 24 hours is not resilience, it is just you generating traffic to nowhere and delaying the moment the customer finds out their config is broken. That third bucket is the one teams skip, and it is the one that turns your retry queue into a landfill.\n\nWhile we are here, two things that are not optional.\n\n**Sign the payload.** Every event goes out with an HMAC of the body plus a timestamp, in a header.\n\nYour customer recomputes it with their shared secret and confirms the event actually came from you.\n\nWithout this, your webhook endpoint is a URL that anybody who guesses it can post fake \"payment succeeded\" events to.\n\nInclude the timestamp *inside* the signed content so a captured request can't be replayed at them next week.\n\n**Assume they will process it twice.** Retries mean at-least-once delivery.\n\nThat is not a flaw you can engineer away, it is the shape of the problem: if your request times out you genuinely cannot tell whether they processed it or not.\n\nSo give every event a stable `id`\n\n, tell your customers to key on it, and document it clearly.\n\nExactly-once delivery is marketing. At-least-once plus idempotency is engineering.\n\nRetries run out. It happens. Their endpoint was down for the whole eight hour retry window and there is nothing more to try.\n\nThe event does not get deleted. It goes to a dead letter queue.\n\nAnd here is the part I really want to land, because it is where most implementations stop one step too early: **a dead letter queue nobody can see is just a slower way of losing data.**\n\nPut a UI on it. A dashboard, in your product, where the customer can see their own failed deliveries.\n\nEvent type, timestamp, attempt count, the actual response body you got back from their server.\n\nThat last one saves so many support tickets, because \"we got `502 Bad Gateway`\n\nfrom your server at 3:04pm\" ends an argument that \"webhooks are broken\" would otherwise stretch across four days.\n\nThen give them a Replay button.\n\nThey fixed their deploy, they click replay, the events go back into the outbox as pending and flow through the exact same pipeline.\n\nBulk replay for a time range, so they can recover a whole outage window in one click.\n\nYou have just converted your worst support conversation into a self-serve action. That is a genuinely good trade.\n\n```\nflowchart LR\n    APP[App writes event<br/>+ business change<br/>in one transaction] --> OUT[(Outbox)]\n    OUT --> DISP[Dispatcher<br/>per-customer caps]\n    DISP --> W[Worker pool]\n    W -->|HMAC signed POST| CUST[Customer endpoint]\n    CUST -->|2xx| DONE[Delivered]\n    CUST -->|5xx / timeout| RETRY[Backoff + jitter]\n    CUST -->|4xx permanent| DLQ[(Dead letter queue)]\n    RETRY --> W\n    RETRY -->|attempts exhausted| DLQ\n    DLQ --> UI[Replay UI]\n    UI -->|customer clicks replay| OUT\n\n    classDef store  fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a\n    classDef proc   fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a\n    classDef ext    fill:#6ea8ff,stroke:#3565bd,color:#1a1a1a\n    classDef bad    fill:#ff9a5c,stroke:#c26a33,color:#1a1a1a\n\n    class OUT,DLQ store\n    class APP,DISP,W,UI,DONE proc\n    class CUST ext\n    class RETRY bad\n```\n\nRead it as one sentence: write it down before you send it, be fair about who you send next, send it signed with a short timeout, retry the failures that deserve retrying, and make the ones that don't visible to the human who can actually fix them.\n\nA few things that don't fit neatly into the versions but will absolutely find you.\n\n**Ordering.** Somebody will ask for it. Ordered delivery per customer means concurrency 1 for that customer, which means one slow response stalls their entire stream. It is a real trade, not a free feature. Usually the better answer is to send a sequence number and let them sort, or send a \"something changed, come fetch it\" ping instead of the state itself.\n\n**Payload size.** Do not put a 4MB object in a webhook. Send the id and the event type, let them call your API for the rest. Thin payloads are cheaper to store in the outbox, cheaper to retry, and they sidestep the awkward question of what happens when the state changed between the event firing and them reading it.\n\n**SSRF.** Customers hand you a URL and you make your servers fetch it. That is textbook server-side request forgery. Resolve the hostname, reject private ranges and link-local addresses, and re-check on redirects, because `http://customer.com/hook`\n\nredirecting to `169.254.169.254`\n\nis somebody trying to read your cloud metadata credentials. [OWASP has the full list](https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html).\n\n**Poison events.** One event that crashes your worker on deserialize will be retried forever and take a worker down with it every single time. Cap attempts on your *own* failures too, not just theirs.\n\n**Meme idea 3** — Template: *They Don't Know* (the guy alone in the corner at a party, thought bubble)\n\nThe POST request is half a day. Genuinely.\n\nThe other 95% is the outbox that keeps the event alive, the dispatcher that stops one customer from ruining everyone's afternoon, the retry classifier that knows the difference between \"try again\" and \"this will never work\", and the replay UI that turns a data loss incident into a button.\n\nNone of that is exotic. It is one table, one dispatcher loop, and a bit of discipline about failure modes.\n\nBut it is the difference between a webhook system your customers trust and one they write defensive polling code around, which is what they will do the second they miss an event and you cannot tell them where it went.\n\nWrite it down first. Everything else follows from that.\n\nIf you have been building on top of a webhook pipeline like this, I'd genuinely like to hear which version you're currently stuck on.\n\nMy guess is version two, and the customer causing the jam is one you can name from memory.\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production code safe without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n**Try LiveReview on your codebase:**", "url": "https://wpnews.pro/news/designing-a-webhook-delivery-system-for-10-million-events-a-day", "canonical_source": "https://dev.to/lovestaco/designing-a-webhook-delivery-system-for-10-million-events-a-day-2p5d", "published_at": "2026-09-02 17:30:40+00:00", "updated_at": "2026-09-02 17:53:58.695045+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Maneshwar", "LiveReview"], "alternates": {"html": "https://wpnews.pro/news/designing-a-webhook-delivery-system-for-10-million-events-a-day", "markdown": "https://wpnews.pro/news/designing-a-webhook-delivery-system-for-10-million-events-a-day.md", "text": "https://wpnews.pro/news/designing-a-webhook-delivery-system-for-10-million-events-a-day.txt", "jsonld": "https://wpnews.pro/news/designing-a-webhook-delivery-system-for-10-million-events-a-day.jsonld"}}