Long-Running AI Jobs: Webhooks You Can Trust A developer built a reliable pipeline for long-running AI jobs using Replicate webhooks, combining signature and timestamp verification, idempotent queueing keyed on the webhook ID, and a reconciler that polls the provider every five minutes to recover results lost when webhook retries fail. The design handles duplicate, replayed, out-of-order and missed webhooks, and copies model output to the developer's own storage before the provider's one-hour deletion window closes. This job finished, and nobody will ever know. A customer asked for a 40-page report, and the model took six minutes. When it finished, your webhook endpoint was in the middle of a deploy and answered 503. The provider retried for about a minute, then gave up. An hour later, the output was deleted. In this article we build long-running AI jobs that never lose a result: a webhook endpoint you can trust, a worker that applies each result exactly once, and a reconciler that catches what the webhooks miss. A model call that takes minutes can't live inside a request. Hold the connection open, and something in between times out. Poll every two seconds, and you burn your rate limit on jobs that aren't done. So you give the provider a webhook URL, and it calls you when the job finishes. But a webhook is a push with no guarantee. It can arrive twice, late, out of order, or never. Four webhooks arrive for the same job. Would you process each one, or not? Decide before reading on. succeeded , a new webhook-id , a valid signature. status=processing , arriving after One: process it. Two: that's a retry. Ignore it, but still answer 200, or the retries keep coming. Three: reject it. The signature is valid, but it's a replay. Four: ignore it. A job never moves backwards. Keep two, three and four in mind: the code below handles each of them. Your API writes a job row first, then asks Replicate to run the model. The webhook endpoint only checks and queues: verify the signature, put the event on a queue, answer in milliseconds. A worker applies the result and copies the output to your own storage, because the provider deletes it after an hour. And a reconciler runs every five minutes, asking the provider about any job that has been running too long. Webhooks are the fast path. The reconciler is the guarantee. Back to the job from the introduction. The API stores the job and creates the prediction. Six minutes later, the webhook hits the deploy: 503. The final retry fails too, and the provider gives up. Before, that was the end. Now, a few minutes later, the reconciler finds a job running too long, asks Replicate, and gets succeeded . The worker copies the report to S3, and the customer gets it. Late, but not lost. → POST /reports job=4471 status=queued replicate.create ........ p 8f2k webhook=…/replicate 14:02:07 webhook ....... 503 deploy in progress 14:03:09 final retry ... 503 · provider gave up 14:05:00 reconciler .... job 4471 running 9 min → GET p 8f2k succeeded → copy output → s3://reports/4471.pdf job 4471 succeeded · 3 min late · 0 lost The timestamp comes first: if it's older than five minutes, reject it. That's webhook three. Then the signature: an HMAC of the webhook ID, the timestamp and the raw body, compared with every signature in the header. Then queue the event, using the webhook ID as the queue's job ID. A retry, like webhook two, lands on the same ID and adds nothing. And answer 200 right away: the provider expects a reply within a few seconds. // webhooks/replicate.ts router.post '/replicate', raw { type: ' / ' } , async req, res = { const id = req.header 'webhook-id' ?? ''; const ts = Number req.header 'webhook-timestamp' ; if Math.abs Date.now / 1000 - ts 300 return res.sendStatus 400 ; const sig = req.header 'webhook-signature' ?? ''; if verify req.body, id, ts, sig return res.sendStatus 401 ; // Same webhook-id on every retry: the queue keeps one job per id await queue.add 'apply', JSON.parse String req.body , { jobId: id } ; res.sendStatus 200 ; // in milliseconds: the worker does the slow part } ; function verify raw: Buffer, id: string, ts: number, header: string { const expected = createHmac 'sha256', SECRET .update ${id}.${ts}.${raw} .digest 'base64' ; const sigs = header.split ' ' .map s = s.split ',' 1 ; return sigs.some s = safeEqual s, expected ; } The worker applies the result in one transaction. Mark the webhook ID as seen; if it was already there, it's a retry, so stop. Then lock the job. If it's already finished, the event is late. That's webhook four: a job never moves backwards. If the prediction succeeded, copy the output to your own storage before it disappears, and finish the job. The reconciler runs every five minutes and checks each job that is still running with the provider, through the same function. The webhook is a hint. The provider is the truth. js // jobs/apply.ts const TERMINAL = 'succeeded', 'failed', 'canceled' ; export const apply = eventId: string | null, p: Prediction = db.tx async tx = { if eventId && await tx.markSeen eventId return; // a retry const job = await tx.jobs.lock p.id ; // SELECT … FOR UPDATE if TERMINAL.includes job.status return; // late: never go back if TERMINAL.includes p.status return; // still running const key = p.status === 'succeeded' ? await copyToS3 p.output, job.id : null; await tx.jobs.finish job.id, p.status, key ; } ; // Every 5 minutes: the webhook is a hint, the provider is the truth export async function reconcile { for const job of await db.jobs.runningFor '5 minutes' { await apply null, await replicate.get job.predictionId ; } } And nobody has to ask where their report is. The numbers are targets for an illustrative scenario, not measurements. ▶️ Watch the video: https://www.youtube.com/watch?v= Arkjh6IaZ4 https://www.youtube.com/watch?v= Arkjh6IaZ4 💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks 🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub confirmation=1 https://www.youtube.com/@NodeNeuralAI?sub confirmation=1