# Your Webhook Endpoint is a Tiny Distributed System > Source: > Published: 2026-09-10 00:07:36+00:00 If you want the Rails implementation version of this, [Webhooks in Rails](https://dev.to/webhooks-in-rails/) goes deeper on verification, durable receipt, idempotency, retries, jobs, testing, and provider-specific behavior, and includes an Agent Companion for repo-aware coding agents. Already have webhook code in an existing Rails application? The [free Webhook Architecture Checkup](https://dev.to/webhooks-in-rails/checkup/) is a repo-aware prompt for tracing the flow you already have and finding the important gaps. Webhook endpoints always seem simple when you build the first version. Add a route, create a controller action, parse some JSON, update a record and return `200`. Pretty standard Rails stuff. Then the real requirements start showing up. You need to verify that Stripe or GitHub actually sent the request. The provider wants a response quickly, so the useful work moves into a background job. The same event arrives twice. A worker dies after doing half of the work. Two related events get processed at the same time. Another event shows up out of order. At some point, that little controller action has picked up a surprising amount of infrastructure around it. Nobody starts by saying, "I need a distributed system for this webhook." You normally get there one completely reasonable requirement at a time. It is still a small system, of course. We are not building Kafka and twelve services here. But once a webhook is production-ready, you have an external trust boundary, durable ingress, asynchronous workers, duplicate delivery, retries, concurrency, ordering problems and a handful of failure states that all need to agree with each other. That is the part of webhooks I find interesting. A tiny HTTP endpoint becomes a pretty good microcosm of a much larger distributed system. A webhook is public ingress into your application, so before doing anything useful with the payload, you need to answer the obvious security question: **did the provider actually send this?** Most providers solve this with a shared secret and a signature over the request body. GitHub, for example, sends an HMAC-SHA256 signature in `X-Hub-Signature-256`. Stripe signs its payload and includes a timestamp in `Stripe-Signature`. The important detail is that those signatures are based on the **raw request body**, not a Ruby hash after Rails has parsed it. So the order here matters: Rails gives us [`request.raw_post`](https://api.rubyonrails.org/classes/ActionDispatch/Request.html#method-i-raw_post) for the first part. A minimal GitHub controller can look something like this: ``` class Webhooks::GithubController < ApplicationController skip_forgery_protection def create raw_body = request.raw_post return head :unauthorized unless valid_signature?(raw_body) payload = JSON.parse(raw_body) # Persist and dispatch the verified event here. head :accepted rescue JSON::ParserError head :bad_request end private def valid_signature?(raw_body) secret = Rails.application.credentials.dig(:github, :webhook_secret) expected = "sha256=" + OpenSSL::HMAC.hexdigest( OpenSSL::Digest.new("sha256"), secret, raw_body ) provided = request.headers["X-Hub-Signature-256"].to_s ActiveSupport::SecurityUtils.secure_compare(expected, provided) end end ``` There are only a few lines here, but each one matters. The signature is compared with Rails' `secure_compare`, the body is not parsed until after verification, and the endpoint skips normal browser CSRF protection because this request is not coming from one of our application's browser sessions. The provider signature is what authenticates this request. For Stripe snapshot events, I would use the SDK instead of rebuilding its signature format myself: ``` raw_body = request.raw_post signature = request.headers["Stripe-Signature"] secret = Rails.application.credentials.dig(:stripe, :webhook_secret) event = Stripe::Webhook.construct_event(raw_body, signature, secret) ``` Stripe's verifier also validates the signed timestamp against its tolerance, which helps with replay protection. Stripe's docs are also very specific about passing the unmodified request body to the verifier. Stripe's newer thin events use a different SDK entry point, so this example is specifically for the snapshot Event objects used by the rest of this article. So even before we process the event, the webhook endpoint already has its own trust boundary and its own security rules. Ok, now that we trust the request, the next problem is how long we keep the provider waiting. The webhook provider is making a normal HTTP request to our application. It does not want to wait while we update a bunch of records, call another API, generate something, send email or do whatever the event actually triggers. GitHub.com expects a `2xx` response within 10 seconds. Stripe's guidance is to return a successful `2xx` before doing complex processing that could time out. That leads to a fairly boring ingress path, which is exactly what I want: ``` verify ↓ validate enough to identify the event ↓ persist a durable receipt ↓ enqueue work ↓ return 2xx ``` Personally, I do not want the request itself provisioning an account, recalculating billing, synchronizing a repository or doing some other multi-step business process. The request should establish that I trust the event and that my application has accepted responsibility for it. New Rails 8 applications use Solid Queue as the default Active Job backend in production. A dispatch job can stay very normal Rails code: ``` python class WebhookDispatchJob < ApplicationJob queue_as :webhooks def perform(delivery_id) delivery = WebhookDelivery.find(delivery_id) # Dispatch to provider/event-specific handling. end end ``` That gets the slow work out of the HTTP request, but just throwing `perform_later` into the controller is not the whole reliability story. If you return a `2xx` and then discover that the event never made it into a durable place, the provider thinks you have the event and you may not actually have it anymore. This is where the acknowledgement boundary becomes important. `2xx` actually mean? I like storing a durable webhook receipt in the application database before telling the provider the event was accepted. The first version does not need to be complicated: ``` class CreateWebhookDeliveries < ActiveRecord::Migration[8.0] def change create_table :webhook_deliveries do |t| t.string :provider, null: false t.string :external_id, null: false t.string :event_type, null: false t.json :payload, null: false t.datetime :processed_at t.datetime :failed_at t.text :last_error t.timestamps end add_index :webhook_deliveries, [:provider, :external_id], unique: true end end ``` With that in place, GitHub ingress can start looking more like this: ``` def create raw_body = request.raw_post return head :unauthorized unless valid_signature?(raw_body) delivery_id = request.headers["X-GitHub-Delivery"].presence event_type = request.headers["X-GitHub-Event"].presence return head :bad_request unless delivery_id && event_type payload = JSON.parse(raw_body) delivery = WebhookDelivery.create_or_find_by!( provider: "github", external_id: delivery_id ) do |record| record.event_type = event_type record.payload = payload end WebhookDispatchJob.perform_later(delivery.id) head :accepted rescue JSON::ParserError head :bad_request end ``` One thing to call out here: the unique database index is doing real work. An `exists?` check followed by `create!` still has a race between those two queries. Two concurrent requests can both pass the check. The unique `(provider, external_id)` constraint is the final guard that prevents two receipt rows for the same provider event. The receipt also gives us somewhere to recover from the slightly awkward gap between inserting application data and enqueuing a job. Solid Queue has an important detail here. In a default Rails 8 production setup, Solid Queue is configured on a separate database from the application's primary database. Creating a `WebhookDelivery` in one database and inserting a Solid Queue job in another is **not one atomic transaction**. In the controller above, `create_or_find_by!` has already finished its transaction before `perform_later` runs, so the receipt is committed before enqueueing starts. If receipt creation and enqueueing happen inside a larger application transaction, Rails gives Active Job `enqueue_after_transaction_commit` to defer the enqueue until that transaction commits: ``` class WebhookDispatchJob < ApplicationJob queue_as :webhooks self.enqueue_after_transaction_commit = true end ``` That prevents work from being enqueued for a transaction that later rolls back. It still does not create one atomic transaction across the application and queue databases. If the receipt commits and the queue insertion fails, I still want that event to be recoverable. A recurring reconciliation job that looks for old, unprocessed receipts is a simple solution and gives you another chance to enqueue the work. Now `2xx` has a much more useful meaning. It does not mean all of the business logic finished. It means the application has durably accepted the event and can recover the remaining work if something goes wrong. This is probably the first webhook problem that surprises people when they have only worked with the happy path. Imagine this: That is not some wild production edge case. Two systems communicated over a network and disagreed about whether the operation finished. Stripe explicitly documents that the same event can be delivered more than once. GitHub gives every delivery an `X-GitHub-Delivery` GUID and keeps that same GUID when a delivery is redelivered. This is why the `(provider, external_id)` unique index is useful. It gives us **delivery idempotency**. Receiving the same provider event over and over does not create a new logical event in our application every time. However, delivery idempotency is not the same as **business idempotency**. Suppose two different webhook events can both lead to the same business operation. Or suppose a job calls an external API successfully and dies before it records `processed_at`. The unique webhook receipt does not stop that external operation from happening again on the retry. For those cases, I like the idempotency boundary to describe the actual operation: ``` class CreateProvisioningOperations < ActiveRecord::Migration[8.0] def change create_table :provisioning_operations do |t| t.references :account, null: false, foreign_key: true t.string :operation_key, null: false t.datetime :completed_at t.timestamps end add_index :provisioning_operations, [:account_id, :operation_key], unique: true end end ``` That `operation_key` could be based on a provider object ID and the action you are performing. If the next external service supports its own idempotency keys, I would use those too when making that request. This is also why I try not to think in terms of "exactly once" webhook processing. We do not control exactly how many times a provider attempts delivery, how many times a job is retried or whether a worker dies at a convenient point. What we can control is whether doing the work again is safe. There is another small wrinkle in the controller example above. Two copies of the same event can both resolve to the same `WebhookDelivery`, and both requests can enqueue `WebhookDispatchJob`. If the job starts with this: ``` return if delivery.processed_at? ``` ...that helps with a later retry, but it does not protect against two workers starting at the same time. Both workers can read `processed_at == nil` before either one updates it. For work that stays entirely inside our database, a row lock can make that state transition explicit: ``` python class WebhookDispatchJob < ApplicationJob queue_as :webhooks def perform(delivery_id) delivery = WebhookDelivery.find(delivery_id) delivery.with_lock do return if delivery.processed_at? Webhooks::Github::Dispatch.call(delivery) delivery.update!(processed_at: Time.current) end end end ``` That example assumes `Dispatch` is doing database work that belongs inside the transaction. I would **not** keep a row lock open while making a slow request to another service. Once the handler has external side effects, that business operation should normally have its own durable idempotency protection instead of assuming a database lock in Rails can control what another system does. So now our tiny webhook endpoint has a concurrency model too. Fun! Once processing moves to Solid Queue, there are two completely separate retry loops to think about: the provider retrying webhook delivery and our application retrying the async processing. For Stripe, the flow looks roughly like this: ``` Stripe ↓ webhook delivery Rails ingress ↓ durable receipt Solid Queue ↓ processing attempt business state ``` If Rails does not return a successful response, Stripe can retry the webhook delivery for up to three days in live mode using exponential backoff. Once Rails has accepted the event, the job can fail independently. Active Job does not automatically retry every failed job for you; retry behavior is something you configure. A transient network failure could use something like: ``` class WebhookDispatchJob < ApplicationJob queue_as :webhooks retry_on Net::OpenTimeout, Net::ReadTimeout, wait: :polynomially_longer, attempts: 5 def perform(delivery_id) # ... end end ``` I would not put every exception behind one giant retry rule. A timeout to another API may be temporary. A malformed payload is not. A missing mapping or a bug in the handler probably needs a fix rather than five delayed attempts at the exact same broken code. Provider behavior is not universal here either. GitHub does **not** automatically redeliver a failed webhook delivery. You can redeliver manually or through the API, but that is very different from Stripe retrying failed deliveries on its own. That is another reason I like owning the durable receipt and having a reconciliation path inside the Rails application. Recovery should not depend entirely on whatever retry policy the provider happens to use. Even after making every individual event safe to retry, you can still get the wrong result if your code assumes events arrive in the order they happened. Stripe does not guarantee event ordering. GitHub also documents that webhook deliveries can arrive in a different order than the underlying events occurred. A billing flow could end up looking like this: ``` subscription.updated subscription.deleted subscription.updated # delayed older event ``` If each handler blindly writes its payload to the local record, that delayed event can move the local state backward. The fix depends on the provider. Sometimes there is a useful sequence or version. Be careful with timestamps: Stripe's snapshot Event `created` value only has second-level precision, and Stripe says not to use it to determine event order. For important state, another option is to treat the webhook as a notification that something changed and then fetch the provider's current canonical object before making a local decision. That read can fail when the object has been deleted or is no longer available, and it is not an ordering guarantee by itself. Concurrent handlers still need a provider-specific version or sequence when one exists, local serialization, or a reconciliation rule. The main point is to make the ordering assumption explicit. "This is the webhook I am processing now" does not automatically mean "this is the newest state." At this point we have duplicate messages, retries, concurrent workers and events showing up out of order. The distributed-system comparison is doing a lot less metaphorical work than it did at the beginning. There is definitely common webhook infrastructure worth sharing, but I would be careful about trying to make Stripe, GitHub, Shopify, Twilio and every other provider look identical too early. Verification happens before trust. Durable receipts can share a lifecycle. Jobs can share dispatch infrastructure. Logging and admin tooling can be normalized. The provider details still matter. Stripe and GitHub alone are enough to show the differences: `Stripe-Signature` format and provides an SDK verifier. GitHub uses `X-Hub-Signature-256` with HMAC-SHA256.`X-GitHub-Delivery` that stays the same across redelivery. I normally prefer a common ingress shape with provider-specific verification and dispatch over a generic abstraction that hides details the application actually needs to know. In Rails, this can stay pretty conventional: ``` Webhooks::GithubController ↓ GithubSignatureVerifier ↓ WebhookDelivery ↓ WebhookDispatchJob ↓ Webhooks::Github::Dispatch ↓ provider/event-specific handler ``` There does not need to be a framework behind every box. The useful part is having an obvious place for each responsibility so that Stripe-specific logic does not slowly leak into GitHub processing and vice versa. `200` is not observability Once the useful work is asynchronous, the HTTP request stops telling us very much about what eventually happened. A successful webhook response can mean the signature passed, a receipt was stored and the work was enqueued while absolutely none of the actual business processing has happened yet. Eventually you need to answer questions about the event lifecycle instead of the HTTP request. Did we receive delivery `abc123`? Was it a duplicate? What event type was it? Did processing ever finish? How many times did it fail? What was the last error? Can I replay it after fixing the bug? This is another place where the `WebhookDelivery` model earns its keep. It gives logs, admin tooling, reconciliation and manual replay a stable record to work from. I usually want enough state to distinguish something along these lines: ``` received → processing → processed ↘ failed ``` You do not need to build a workflow engine for this. A few timestamps, an error field and a clear job boundary may be plenty. But once processing can happen seconds or minutes after the request, "the endpoint returned 200" is not a useful debugging answer anymore. Let's look back at everything our original controller picked up along the way. We now have an untrusted network boundary, cryptographic verification, a durable message receipt, database uniqueness, an asynchronous queue, concurrent workers, duplicate delivery, an internal retry policy, a provider retry policy, out-of-order events, business idempotency, failure state, reconciliation, replay and observability. None of those are especially exotic problems by themselves. The interesting part is how many of them get packed into a feature that started as `POST /webhooks/stripe`. This does **not** mean the first webhook endpoint needs fifteen classes and a homegrown framework. I would still start pretty small. I just want the important boundaries to be obvious: ``` POST /webhooks/:provider ↓ verify the raw payload ↓ persist one durable receipt ↓ enqueue Solid Queue work ↓ provider-specific handler ↓ idempotent business operation ↓ record the outcome ``` From there, add complexity when a real failure mode calls for it. The controller can stay boring. That is probably the best outcome. The complicated part was never parsing the JSON. It was accepting work from another system when you do not control how many times the message arrives, when it arrives, what order it arrives in or which process eventually finishes it. That is a lot hiding behind one Rails controller action.