cd /news/ai-infrastructure/long-running-ai-jobs-webhooks-you-ca… · home › topics › ai-infrastructure › article
[ARTICLE · art-170644] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Long-Running AI Jobs: Webhooks You Can Trust

A developer built a reliable pipeline for long-running AI jobs using Replicate webhooks, combining signature and timestamp verification, idempotent queueing keyed on the webhook ID, and a reconciler that polls the provider every five minutes to recover results lost when webhook retries fail. The design handles duplicate, replayed, out-of-order and missed webhooks, and copies model output to the developer's own storage before the provider's one-hour deletion window closes.

by read4 min views1 publishedNov 16, 2026

This job finished, and nobody will ever know. A customer asked for a 40-page report, and the model took six minutes. When it finished, your webhook endpoint was in the middle of a deploy and answered 503. The provider retried for about a minute, then gave up. An hour later, the output was deleted.

In this article we build long-running AI jobs that never lose a result: a webhook endpoint you can trust, a worker that applies each result exactly once, and a reconciler that catches what the webhooks miss.

A model call that takes minutes can't live inside a request. Hold the connection open, and something in between times out. Poll every two seconds, and you burn your rate limit on jobs that aren't done. So you give the provider a webhook URL, and it calls you when the job finishes.

But a webhook is a push with no guarantee. It can arrive twice, late, out of order, or never.

Four webhooks arrive for the same job. Would you process each one, or not? Decide before reading on.

succeeded, a new webhook-id, a valid signature. status=processing, arriving after One: process it.

Two: that's a retry. Ignore it, but still answer 200, or the retries keep coming.

Three: reject it. The signature is valid, but it's a replay.

Four: ignore it. A job never moves backwards.

Keep two, three and four in mind: the code below handles each of them.

Your API writes a job row first, then asks Replicate to run the model. The webhook endpoint only checks and queues: verify the signature, put the event on a queue, answer in milliseconds. A worker applies the result and copies the output to your own storage, because the provider deletes it after an hour.

And a reconciler runs every five minutes, asking the provider about any job that has been running too long. Webhooks are the fast path. The reconciler is the guarantee.

Back to the job from the introduction. The API stores the job and creates the prediction. Six minutes later, the webhook hits the deploy: 503. The final retry fails too, and the provider gives up. Before, that was the end. Now, a few minutes later, the reconciler finds a job running too long, asks Replicate, and gets succeeded.

The worker copies the report to S3, and the customer gets it. Late, but not lost.

→ POST /reports   job=4471  status=queued
  replicate.create ........ p_8f2k  webhook=…/replicate
  14:02:07  webhook ....... 503  (deploy in progress)
  14:03:09  final retry ... 503  · provider gave up
  14:05:00  reconciler .... job 4471 running 9 min → GET p_8f2k
  succeeded → copy output → s3://reports/4471.pdf
job 4471 succeeded · 3 min late · 0 lost

The timestamp comes first: if it's older than five minutes, reject it. That's webhook three. Then the signature: an HMAC of the webhook ID, the timestamp and the raw body, compared with every signature in the header. Then queue the event, using the webhook ID as the queue's job ID.

A retry, like webhook two, lands on the same ID and adds nothing. And answer 200 right away: the provider expects a reply within a few seconds.

// webhooks/replicate.ts
router.post('/replicate', raw({ type: '*/*' }), async (req, res) => {
  const id = req.header('webhook-id') ?? '';
  const ts = Number(req.header('webhook-timestamp'));
  if (Math.abs(Date.now() / 1000 - ts) > 300) return res.sendStatus(400);

  const sig = req.header('webhook-signature') ?? '';
  if (!verify(req.body, id, ts, sig)) return res.sendStatus(401);

  // Same webhook-id on every retry: the queue keeps one job per id
  await queue.add('apply', JSON.parse(String(req.body)), { jobId: id });
  res.sendStatus(200); // in milliseconds: the worker does the slow part
});

function verify(raw: Buffer, id: string, ts: number, header: string) {
  const expected = createHmac('sha256', SECRET)
    .update(`${id}.${ts}.${raw}`).digest('base64');
  const sigs = header.split(' ').map((s) => s.split(',')[1]);
  return sigs.some((s) => safeEqual(s, expected));
}

The worker applies the result in one transaction. Mark the webhook ID as seen; if it was already there, it's a retry, so stop. Then lock the job. If it's already finished, the event is late. That's webhook four: a job never moves backwards. If the prediction succeeded, copy the output to your own storage before it disappears, and finish the job.

The reconciler runs every five minutes and checks each job that is still running with the provider, through the same function. The webhook is a hint. The provider is the truth.

// jobs/apply.ts
const TERMINAL = ['succeeded', 'failed', 'canceled'];

export const apply = (eventId: string | null, p: Prediction) =>
  db.tx(async (tx) => {
    if (eventId && !(await tx.markSeen(eventId))) return; // a retry
    const job = await tx.jobs.lock(p.id); // SELECT … FOR UPDATE
    if (TERMINAL.includes(job.status)) return; // late: never go back
    if (!TERMINAL.includes(p.status)) return; // still running

    const key =
      p.status === 'succeeded' ? await copyToS3(p.output, job.id) : null;
    await tx.jobs.finish(job.id, p.status, key);
  });

// Every 5 minutes: the webhook is a hint, the provider is the truth
export async function reconcile() {
  for (const job of await db.jobs.runningFor('5 minutes')) {
    await apply(null, await replicate.get(job.predictionId));
  }
}

And nobody has to ask where their report is.

The numbers are targets for an illustrative scenario, not measurements.

▶️ Watch the video: https://www.youtube.com/watch?v=_Arkjh6IaZ4

💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks

🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @replicate 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/long-running-ai-jobs…] indexed:0 read:4min 2026-11-16 · —