{"slug": "long-running-ai-jobs-webhooks-you-can-trust", "title": "Long-Running AI Jobs: Webhooks You Can Trust", "summary": "A developer built a reliable pipeline for long-running AI jobs using Replicate webhooks, combining signature and timestamp verification, idempotent queueing keyed on the webhook ID, and a reconciler that polls the provider every five minutes to recover results lost when webhook retries fail. The design handles duplicate, replayed, out-of-order and missed webhooks, and copies model output to the developer's own storage before the provider's one-hour deletion window closes.", "body_md": "This job finished, and nobody will ever know. A customer asked for a 40-page report, and the model took six minutes. When it finished, your webhook endpoint was in the middle of a deploy and answered 503. The provider retried for about a minute, then gave up. An hour later, the output was deleted.\n\nIn this article we build long-running AI jobs that never lose a result: a webhook endpoint you can trust, a worker that applies each result exactly once, and a reconciler that catches what the webhooks miss.\n\nA model call that takes minutes can't live inside a request. Hold the connection open, and something in between times out. Poll every two seconds, and you burn your rate limit on jobs that aren't done. So you give the provider a webhook URL, and it calls you when the job finishes.\n\nBut a webhook is a push with no guarantee. It can arrive twice, late, out of order, or never.\n\nFour webhooks arrive for the same job. Would you process each one, or not? Decide before reading on.\n\n`succeeded`, a new `webhook-id`, a valid signature.` status=processing`, arriving after **One:** process it.\n\n**Two:** that's a retry. Ignore it, but still answer 200, or the retries keep coming.\n\n**Three:** reject it. The signature is valid, but it's a replay.\n\n**Four:** ignore it. A job never moves backwards.\n\nKeep two, three and four in mind: the code below handles each of them.\n\nYour API writes a job row first, then asks Replicate to run the model. The webhook endpoint only checks and queues: verify the signature, put the event on a queue, answer in milliseconds. A worker applies the result and copies the output to your own storage, because the provider deletes it after an hour.\n\nAnd a reconciler runs every five minutes, asking the provider about any job that has been running too long. Webhooks are the fast path. The reconciler is the guarantee.\n\nBack to the job from the introduction. The API stores the job and creates the prediction. Six minutes later, the webhook hits the deploy: 503. The final retry fails too, and the provider gives up. Before, that was the end. Now, a few minutes later, the reconciler finds a job running too long, asks Replicate, and gets `succeeded`.\n\nThe worker copies the report to S3, and the customer gets it. Late, but not lost.\n\n```\n→ POST /reports   job=4471  status=queued\n  replicate.create ........ p_8f2k  webhook=…/replicate\n  14:02:07  webhook ....... 503  (deploy in progress)\n  14:03:09  final retry ... 503  · provider gave up\n  14:05:00  reconciler .... job 4471 running 9 min → GET p_8f2k\n  succeeded → copy output → s3://reports/4471.pdf\njob 4471 succeeded · 3 min late · 0 lost\n```\n\nThe timestamp comes first: if it's older than five minutes, reject it. That's webhook three. Then the signature: an HMAC of the webhook ID, the timestamp and the raw body, compared with every signature in the header. Then queue the event, using the webhook ID as the queue's job ID.\n\nA retry, like webhook two, lands on the same ID and adds nothing. And answer 200 right away: the provider expects a reply within a few seconds.\n\n```\n// webhooks/replicate.ts\nrouter.post('/replicate', raw({ type: '*/*' }), async (req, res) => {\n  const id = req.header('webhook-id') ?? '';\n  const ts = Number(req.header('webhook-timestamp'));\n  if (Math.abs(Date.now() / 1000 - ts) > 300) return res.sendStatus(400);\n\n  const sig = req.header('webhook-signature') ?? '';\n  if (!verify(req.body, id, ts, sig)) return res.sendStatus(401);\n\n  // Same webhook-id on every retry: the queue keeps one job per id\n  await queue.add('apply', JSON.parse(String(req.body)), { jobId: id });\n  res.sendStatus(200); // in milliseconds: the worker does the slow part\n});\n\nfunction verify(raw: Buffer, id: string, ts: number, header: string) {\n  const expected = createHmac('sha256', SECRET)\n    .update(`${id}.${ts}.${raw}`).digest('base64');\n  const sigs = header.split(' ').map((s) => s.split(',')[1]);\n  return sigs.some((s) => safeEqual(s, expected));\n}\n```\n\nThe worker applies the result in one transaction. Mark the webhook ID as seen; if it was already there, it's a retry, so stop. Then lock the job. If it's already finished, the event is late. That's webhook four: a job never moves backwards. If the prediction succeeded, copy the output to your own storage before it disappears, and finish the job.\n\nThe reconciler runs every five minutes and checks each job that is still running with the provider, through the same function. The webhook is a hint. The provider is the truth.\n\n``` js\n// jobs/apply.ts\nconst TERMINAL = ['succeeded', 'failed', 'canceled'];\n\nexport const apply = (eventId: string | null, p: Prediction) =>\n  db.tx(async (tx) => {\n    if (eventId && !(await tx.markSeen(eventId))) return; // a retry\n    const job = await tx.jobs.lock(p.id); // SELECT … FOR UPDATE\n    if (TERMINAL.includes(job.status)) return; // late: never go back\n    if (!TERMINAL.includes(p.status)) return; // still running\n\n    const key =\n      p.status === 'succeeded' ? await copyToS3(p.output, job.id) : null;\n    await tx.jobs.finish(job.id, p.status, key);\n  });\n\n// Every 5 minutes: the webhook is a hint, the provider is the truth\nexport async function reconcile() {\n  for (const job of await db.jobs.runningFor('5 minutes')) {\n    await apply(null, await replicate.get(job.predictionId));\n  }\n}\n```\n\nAnd nobody has to ask where their report is.\n\n*The numbers are targets for an illustrative scenario, not measurements.*\n\n▶️ Watch the video: [https://www.youtube.com/watch?v=_Arkjh6IaZ4](https://www.youtube.com/watch?v=_Arkjh6IaZ4)\n\n💻 Full code: [https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks](https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks)\n\n🔔 More enterprise AI architecture every week: [https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1](https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1)", "url": "https://wpnews.pro/news/long-running-ai-jobs-webhooks-you-can-trust", "canonical_source": "https://dev.to/nodeneuralai/long-running-ai-jobs-webhooks-you-can-trust-39b8", "published_at": "2026-11-16 15:10:48+00:00", "updated_at": "2026-11-16 15:22:09.842995+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "developer-tools", "mlops"], "entities": ["Replicate", "Amazon S3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/long-running-ai-jobs-webhooks-you-can-trust", "markdown": "https://wpnews.pro/news/long-running-ai-jobs-webhooks-you-can-trust.md", "text": "https://wpnews.pro/news/long-running-ai-jobs-webhooks-you-can-trust.txt", "jsonld": "https://wpnews.pro/news/long-running-ai-jobs-webhooks-you-can-trust.jsonld"}}