{"slug": "your-webhook-endpoint-is-a-tiny-distributed-system", "title": "Your Webhook Endpoint is a Tiny Distributed System", "summary": "A developer outlines how a seemingly simple webhook endpoint evolves into a small distributed system, requiring signature verification, durable ingress, asynchronous workers, and handling of duplicate delivery, retries, concurrency, and ordering. The writeup details a Rails implementation using raw request body HMAC-SHA256 verification for GitHub and the Stripe SDK for Stripe events, emphasizing that signatures must be checked before parsing the payload.", "body_md": "If you want the Rails implementation version of this, [Webhooks in Rails](https://dev.to/webhooks-in-rails/) goes deeper on verification, durable receipt, idempotency, retries, jobs, testing, and provider-specific behavior, and includes an Agent Companion for repo-aware coding agents.\n\nAlready have webhook code in an existing Rails application? The [free Webhook Architecture Checkup](https://dev.to/webhooks-in-rails/checkup/) is a repo-aware prompt for tracing the flow you already have and finding the important gaps.\n\nWebhook endpoints always seem simple when you build the first version.\n\nAdd a route, create a controller action, parse some JSON, update a record and return `200`. Pretty standard Rails stuff.\n\nThen the real requirements start showing up. You need to verify that Stripe or GitHub actually sent the request. The provider wants a response quickly, so the useful work moves into a background job. The same event arrives twice. A worker dies after doing half of the work. Two related events get processed at the same time. Another event shows up out of order.\n\nAt some point, that little controller action has picked up a surprising amount of infrastructure around it.\n\nNobody starts by saying, \"I need a distributed system for this webhook.\" You normally get there one completely reasonable requirement at a time.\n\nIt is still a small system, of course. We are not building Kafka and twelve services here. But once a webhook is production-ready, you have an external trust boundary, durable ingress, asynchronous workers, duplicate delivery, retries, concurrency, ordering problems and a handful of failure states that all need to agree with each other.\n\nThat is the part of webhooks I find interesting. A tiny HTTP endpoint becomes a pretty good microcosm of a much larger distributed system.\n\nA webhook is public ingress into your application, so before doing anything useful with the payload, you need to answer the obvious security question: **did the provider actually send this?**\n\nMost providers solve this with a shared secret and a signature over the request body. GitHub, for example, sends an HMAC-SHA256 signature in `X-Hub-Signature-256`. Stripe signs its payload and includes a timestamp in `Stripe-Signature`.\n\nThe important detail is that those signatures are based on the **raw request body**, not a Ruby hash after Rails has parsed it.\n\nSo the order here matters:\n\nRails gives us [`request.raw_post`](https://api.rubyonrails.org/classes/ActionDispatch/Request.html#method-i-raw_post) for the first part.\n\nA minimal GitHub controller can look something like this:\n\n```\nclass Webhooks::GithubController < ApplicationController\n  skip_forgery_protection\n\n  def create\n    raw_body = request.raw_post\n\n    return head :unauthorized unless valid_signature?(raw_body)\n\n    payload = JSON.parse(raw_body)\n\n    # Persist and dispatch the verified event here.\n\n    head :accepted\n  rescue JSON::ParserError\n    head :bad_request\n  end\n\n  private\n    def valid_signature?(raw_body)\n      secret = Rails.application.credentials.dig(:github, :webhook_secret)\n\n      expected = \"sha256=\" + OpenSSL::HMAC.hexdigest(\n        OpenSSL::Digest.new(\"sha256\"),\n        secret,\n        raw_body\n      )\n\n      provided = request.headers[\"X-Hub-Signature-256\"].to_s\n\n      ActiveSupport::SecurityUtils.secure_compare(expected, provided)\n    end\nend\n```\n\nThere are only a few lines here, but each one matters. The signature is compared with Rails' `secure_compare`, the body is not parsed until after verification, and the endpoint skips normal browser CSRF protection because this request is not coming from one of our application's browser sessions. The provider signature is what authenticates this request.\n\nFor Stripe snapshot events, I would use the SDK instead of rebuilding its signature format myself:\n\n```\nraw_body = request.raw_post\nsignature = request.headers[\"Stripe-Signature\"]\nsecret = Rails.application.credentials.dig(:stripe, :webhook_secret)\n\nevent = Stripe::Webhook.construct_event(raw_body, signature, secret)\n```\n\nStripe's verifier also validates the signed timestamp against its tolerance, which helps with replay protection. Stripe's docs are also very specific about passing the unmodified request body to the verifier.\n\nStripe's newer thin events use a different SDK entry point, so this example is specifically for the snapshot Event objects used by the rest of this article.\n\nSo even before we process the event, the webhook endpoint already has its own trust boundary and its own security rules.\n\nOk, now that we trust the request, the next problem is how long we keep the provider waiting.\n\nThe webhook provider is making a normal HTTP request to our application. It does not want to wait while we update a bunch of records, call another API, generate something, send email or do whatever the event actually triggers.\n\nGitHub.com expects a `2xx` response within 10 seconds. Stripe's guidance is to return a successful `2xx` before doing complex processing that could time out.\n\nThat leads to a fairly boring ingress path, which is exactly what I want:\n\n```\nverify\n  ↓\nvalidate enough to identify the event\n  ↓\npersist a durable receipt\n  ↓\nenqueue work\n  ↓\nreturn 2xx\n```\n\nPersonally, I do not want the request itself provisioning an account, recalculating billing, synchronizing a repository or doing some other multi-step business process. The request should establish that I trust the event and that my application has accepted responsibility for it.\n\nNew Rails 8 applications use Solid Queue as the default Active Job backend in production. A dispatch job can stay very normal Rails code:\n\n``` python\nclass WebhookDispatchJob < ApplicationJob\n  queue_as :webhooks\n\n  def perform(delivery_id)\n    delivery = WebhookDelivery.find(delivery_id)\n\n    # Dispatch to provider/event-specific handling.\n  end\nend\n```\n\nThat gets the slow work out of the HTTP request, but just throwing `perform_later` into the controller is not the whole reliability story.\n\nIf you return a `2xx` and then discover that the event never made it into a durable place, the provider thinks you have the event and you may not actually have it anymore.\n\nThis is where the acknowledgement boundary becomes important.\n\n`2xx` actually mean?\nI like storing a durable webhook receipt in the application database before telling the provider the event was accepted.\n\nThe first version does not need to be complicated:\n\n```\nclass CreateWebhookDeliveries < ActiveRecord::Migration[8.0]\n  def change\n    create_table :webhook_deliveries do |t|\n      t.string :provider, null: false\n      t.string :external_id, null: false\n      t.string :event_type, null: false\n      t.json :payload, null: false\n      t.datetime :processed_at\n      t.datetime :failed_at\n      t.text :last_error\n      t.timestamps\n    end\n\n    add_index :webhook_deliveries,\n      [:provider, :external_id],\n      unique: true\n  end\nend\n```\n\nWith that in place, GitHub ingress can start looking more like this:\n\n```\ndef create\n  raw_body = request.raw_post\n  return head :unauthorized unless valid_signature?(raw_body)\n\n  delivery_id = request.headers[\"X-GitHub-Delivery\"].presence\n  event_type = request.headers[\"X-GitHub-Event\"].presence\n\n  return head :bad_request unless delivery_id && event_type\n\n  payload = JSON.parse(raw_body)\n\n  delivery = WebhookDelivery.create_or_find_by!(\n    provider: \"github\",\n    external_id: delivery_id\n  ) do |record|\n    record.event_type = event_type\n    record.payload = payload\n  end\n\n  WebhookDispatchJob.perform_later(delivery.id)\n\n  head :accepted\nrescue JSON::ParserError\n  head :bad_request\nend\n```\n\nOne thing to call out here: the unique database index is doing real work. An `exists?` check followed by `create!` still has a race between those two queries. Two concurrent requests can both pass the check. The unique `(provider, external_id)` constraint is the final guard that prevents two receipt rows for the same provider event.\n\nThe receipt also gives us somewhere to recover from the slightly awkward gap between inserting application data and enqueuing a job.\n\nSolid Queue has an important detail here. In a default Rails 8 production setup, Solid Queue is configured on a separate database from the application's primary database. Creating a `WebhookDelivery` in one database and inserting a Solid Queue job in another is **not one atomic transaction**.\n\nIn the controller above, `create_or_find_by!` has already finished its transaction before `perform_later` runs, so the receipt is committed before enqueueing starts. If receipt creation and enqueueing happen inside a larger application transaction, Rails gives Active Job `enqueue_after_transaction_commit` to defer the enqueue until that transaction commits:\n\n```\nclass WebhookDispatchJob < ApplicationJob\n  queue_as :webhooks\n  self.enqueue_after_transaction_commit = true\nend\n```\n\nThat prevents work from being enqueued for a transaction that later rolls back. It still does not create one atomic transaction across the application and queue databases.\n\nIf the receipt commits and the queue insertion fails, I still want that event to be recoverable. A recurring reconciliation job that looks for old, unprocessed receipts is a simple solution and gives you another chance to enqueue the work.\n\nNow `2xx` has a much more useful meaning. It does not mean all of the business logic finished. It means the application has durably accepted the event and can recover the remaining work if something goes wrong.\n\nThis is probably the first webhook problem that surprises people when they have only worked with the happy path.\n\nImagine this:\n\nThat is not some wild production edge case. Two systems communicated over a network and disagreed about whether the operation finished.\n\nStripe explicitly documents that the same event can be delivered more than once. GitHub gives every delivery an `X-GitHub-Delivery` GUID and keeps that same GUID when a delivery is redelivered.\n\nThis is why the `(provider, external_id)` unique index is useful. It gives us **delivery idempotency**. Receiving the same provider event over and over does not create a new logical event in our application every time.\n\nHowever, delivery idempotency is not the same as **business idempotency**.\n\nSuppose two different webhook events can both lead to the same business operation. Or suppose a job calls an external API successfully and dies before it records `processed_at`. The unique webhook receipt does not stop that external operation from happening again on the retry.\n\nFor those cases, I like the idempotency boundary to describe the actual operation:\n\n```\nclass CreateProvisioningOperations < ActiveRecord::Migration[8.0]\n  def change\n    create_table :provisioning_operations do |t|\n      t.references :account, null: false, foreign_key: true\n      t.string :operation_key, null: false\n      t.datetime :completed_at\n      t.timestamps\n    end\n\n    add_index :provisioning_operations,\n      [:account_id, :operation_key],\n      unique: true\n  end\nend\n```\n\nThat `operation_key` could be based on a provider object ID and the action you are performing. If the next external service supports its own idempotency keys, I would use those too when making that request.\n\nThis is also why I try not to think in terms of \"exactly once\" webhook processing. We do not control exactly how many times a provider attempts delivery, how many times a job is retried or whether a worker dies at a convenient point.\n\nWhat we can control is whether doing the work again is safe.\n\nThere is another small wrinkle in the controller example above.\n\nTwo copies of the same event can both resolve to the same `WebhookDelivery`, and both requests can enqueue `WebhookDispatchJob`.\n\nIf the job starts with this:\n\n```\nreturn if delivery.processed_at?\n```\n\n...that helps with a later retry, but it does not protect against two workers starting at the same time. Both workers can read `processed_at == nil` before either one updates it.\n\nFor work that stays entirely inside our database, a row lock can make that state transition explicit:\n\n``` python\nclass WebhookDispatchJob < ApplicationJob\n  queue_as :webhooks\n\n  def perform(delivery_id)\n    delivery = WebhookDelivery.find(delivery_id)\n\n    delivery.with_lock do\n      return if delivery.processed_at?\n\n      Webhooks::Github::Dispatch.call(delivery)\n      delivery.update!(processed_at: Time.current)\n    end\n  end\nend\n```\n\nThat example assumes `Dispatch` is doing database work that belongs inside the transaction.\n\nI would **not** keep a row lock open while making a slow request to another service. Once the handler has external side effects, that business operation should normally have its own durable idempotency protection instead of assuming a database lock in Rails can control what another system does.\n\nSo now our tiny webhook endpoint has a concurrency model too. Fun!\n\nOnce processing moves to Solid Queue, there are two completely separate retry loops to think about: the provider retrying webhook delivery and our application retrying the async processing.\n\nFor Stripe, the flow looks roughly like this:\n\n```\nStripe\n  ↓ webhook delivery\nRails ingress\n  ↓ durable receipt\nSolid Queue\n  ↓ processing attempt\nbusiness state\n```\n\nIf Rails does not return a successful response, Stripe can retry the webhook delivery for up to three days in live mode using exponential backoff.\n\nOnce Rails has accepted the event, the job can fail independently. Active Job does not automatically retry every failed job for you; retry behavior is something you configure. A transient network failure could use something like:\n\n```\nclass WebhookDispatchJob < ApplicationJob\n  queue_as :webhooks\n\n  retry_on Net::OpenTimeout,\n    Net::ReadTimeout,\n    wait: :polynomially_longer,\n    attempts: 5\n\n  def perform(delivery_id)\n    # ...\n  end\nend\n```\n\nI would not put every exception behind one giant retry rule. A timeout to another API may be temporary. A malformed payload is not. A missing mapping or a bug in the handler probably needs a fix rather than five delayed attempts at the exact same broken code.\n\nProvider behavior is not universal here either. GitHub does **not** automatically redeliver a failed webhook delivery. You can redeliver manually or through the API, but that is very different from Stripe retrying failed deliveries on its own.\n\nThat is another reason I like owning the durable receipt and having a reconciliation path inside the Rails application. Recovery should not depend entirely on whatever retry policy the provider happens to use.\n\nEven after making every individual event safe to retry, you can still get the wrong result if your code assumes events arrive in the order they happened.\n\nStripe does not guarantee event ordering. GitHub also documents that webhook deliveries can arrive in a different order than the underlying events occurred.\n\nA billing flow could end up looking like this:\n\n```\nsubscription.updated\nsubscription.deleted\nsubscription.updated   # delayed older event\n```\n\nIf each handler blindly writes its payload to the local record, that delayed event can move the local state backward.\n\nThe fix depends on the provider. Sometimes there is a useful sequence or version. Be careful with timestamps: Stripe's snapshot Event `created` value only has second-level precision, and Stripe says not to use it to determine event order.\n\nFor important state, another option is to treat the webhook as a notification that something changed and then fetch the provider's current canonical object before making a local decision. That read can fail when the object has been deleted or is no longer available, and it is not an ordering guarantee by itself. Concurrent handlers still need a provider-specific version or sequence when one exists, local serialization, or a reconciliation rule.\n\nThe main point is to make the ordering assumption explicit. \"This is the webhook I am processing now\" does not automatically mean \"this is the newest state.\"\n\nAt this point we have duplicate messages, retries, concurrent workers and events showing up out of order. The distributed-system comparison is doing a lot less metaphorical work than it did at the beginning.\n\nThere is definitely common webhook infrastructure worth sharing, but I would be careful about trying to make Stripe, GitHub, Shopify, Twilio and every other provider look identical too early.\n\nVerification happens before trust. Durable receipts can share a lifecycle. Jobs can share dispatch infrastructure. Logging and admin tooling can be normalized.\n\nThe provider details still matter.\n\nStripe and GitHub alone are enough to show the differences:\n\n`Stripe-Signature` format and provides an SDK verifier. GitHub uses `X-Hub-Signature-256` with HMAC-SHA256.`X-GitHub-Delivery` that stays the same across redelivery.\nI normally prefer a common ingress shape with provider-specific verification and dispatch over a generic abstraction that hides details the application actually needs to know.\n\nIn Rails, this can stay pretty conventional:\n\n```\nWebhooks::GithubController\n        ↓\nGithubSignatureVerifier\n        ↓\nWebhookDelivery\n        ↓\nWebhookDispatchJob\n        ↓\nWebhooks::Github::Dispatch\n        ↓\nprovider/event-specific handler\n```\n\nThere does not need to be a framework behind every box. The useful part is having an obvious place for each responsibility so that Stripe-specific logic does not slowly leak into GitHub processing and vice versa.\n\n`200` is not observability\nOnce the useful work is asynchronous, the HTTP request stops telling us very much about what eventually happened.\n\nA successful webhook response can mean the signature passed, a receipt was stored and the work was enqueued while absolutely none of the actual business processing has happened yet.\n\nEventually you need to answer questions about the event lifecycle instead of the HTTP request.\n\nDid we receive delivery `abc123`? Was it a duplicate? What event type was it? Did processing ever finish? How many times did it fail? What was the last error? Can I replay it after fixing the bug?\n\nThis is another place where the `WebhookDelivery` model earns its keep. It gives logs, admin tooling, reconciliation and manual replay a stable record to work from.\n\nI usually want enough state to distinguish something along these lines:\n\n```\nreceived → processing → processed\n                  ↘ failed\n```\n\nYou do not need to build a workflow engine for this. A few timestamps, an error field and a clear job boundary may be plenty. But once processing can happen seconds or minutes after the request, \"the endpoint returned 200\" is not a useful debugging answer anymore.\n\nLet's look back at everything our original controller picked up along the way.\n\nWe now have an untrusted network boundary, cryptographic verification, a durable message receipt, database uniqueness, an asynchronous queue, concurrent workers, duplicate delivery, an internal retry policy, a provider retry policy, out-of-order events, business idempotency, failure state, reconciliation, replay and observability.\n\nNone of those are especially exotic problems by themselves. The interesting part is how many of them get packed into a feature that started as `POST /webhooks/stripe`.\n\nThis does **not** mean the first webhook endpoint needs fifteen classes and a homegrown framework. I would still start pretty small. I just want the important boundaries to be obvious:\n\n```\nPOST /webhooks/:provider\n        ↓\nverify the raw payload\n        ↓\npersist one durable receipt\n        ↓\nenqueue Solid Queue work\n        ↓\nprovider-specific handler\n        ↓\nidempotent business operation\n        ↓\nrecord the outcome\n```\n\nFrom there, add complexity when a real failure mode calls for it.\n\nThe controller can stay boring. That is probably the best outcome.\n\nThe complicated part was never parsing the JSON. It was accepting work from another system when you do not control how many times the message arrives, when it arrives, what order it arrives in or which process eventually finishes it.\n\nThat is a lot hiding behind one Rails controller action.", "url": "https://wpnews.pro/news/your-webhook-endpoint-is-a-tiny-distributed-system", "canonical_source": "https://dev.to/rob__race/your-webhook-endpoint-is-a-tiny-distributed-system-p71", "published_at": "2026-09-10 00:07:36+00:00", "updated_at": "2026-09-10 00:49:16.159779+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["Rails", "GitHub", "Stripe", "OpenSSL", "ActiveSupport"], "alternates": {"html": "https://wpnews.pro/news/your-webhook-endpoint-is-a-tiny-distributed-system", "markdown": "https://wpnews.pro/news/your-webhook-endpoint-is-a-tiny-distributed-system.md", "text": "https://wpnews.pro/news/your-webhook-endpoint-is-a-tiny-distributed-system.txt", "jsonld": "https://wpnews.pro/news/your-webhook-endpoint-is-a-tiny-distributed-system.jsonld"}}