This job finished, and nobody will ever know. A customer asked for a 40-page report, and the model took six minutes. When it finished, your webhook endpoint was in the middle of a deploy and answered 503. The provider retried for about a minute, then gave up. An hour later, the output was deleted.
In this article we build long-running AI jobs that never lose a result: a webhook endpoint you can trust, a worker that applies each result exactly once, and a reconciler that catches what the webhooks miss.
A model call that takes minutes can't live inside a request. Hold the connection open, and something in between times out. Poll every two seconds, and you burn your rate limit on jobs that aren't done. So you give the provider a webhook URL, and it calls you when the job finishes.
But a webhook is a push with no guarantee. It can arrive twice, late, out of order, or never.
Four webhooks arrive for the same job. Would you process each one, or not? Decide before reading on.
succeeded, a new webhook-id, a valid signature. status=processing, arriving after One: process it.
Two: that's a retry. Ignore it, but still answer 200, or the retries keep coming.
Three: reject it. The signature is valid, but it's a replay.
Four: ignore it. A job never moves backwards.
Keep two, three and four in mind: the code below handles each of them.
Your API writes a job row first, then asks Replicate to run the model. The webhook endpoint only checks and queues: verify the signature, put the event on a queue, answer in milliseconds. A worker applies the result and copies the output to your own storage, because the provider deletes it after an hour.
And a reconciler runs every five minutes, asking the provider about any job that has been running too long. Webhooks are the fast path. The reconciler is the guarantee.
Back to the job from the introduction. The API stores the job and creates the prediction. Six minutes later, the webhook hits the deploy: 503. The final retry fails too, and the provider gives up. Before, that was the end. Now, a few minutes later, the reconciler finds a job running too long, asks Replicate, and gets succeeded.
The worker copies the report to S3, and the customer gets it. Late, but not lost.
→ POST /reports job=4471 status=queued
replicate.create ........ p_8f2k webhook=…/replicate
14:02:07 webhook ....... 503 (deploy in progress)
14:03:09 final retry ... 503 · provider gave up
14:05:00 reconciler .... job 4471 running 9 min → GET p_8f2k
succeeded → copy output → s3://reports/4471.pdf
job 4471 succeeded · 3 min late · 0 lost
The timestamp comes first: if it's older than five minutes, reject it. That's webhook three. Then the signature: an HMAC of the webhook ID, the timestamp and the raw body, compared with every signature in the header. Then queue the event, using the webhook ID as the queue's job ID.
A retry, like webhook two, lands on the same ID and adds nothing. And answer 200 right away: the provider expects a reply within a few seconds.
// webhooks/replicate.ts
router.post('/replicate', raw({ type: '*/*' }), async (req, res) => {
const id = req.header('webhook-id') ?? '';
const ts = Number(req.header('webhook-timestamp'));
if (Math.abs(Date.now() / 1000 - ts) > 300) return res.sendStatus(400);
const sig = req.header('webhook-signature') ?? '';
if (!verify(req.body, id, ts, sig)) return res.sendStatus(401);
// Same webhook-id on every retry: the queue keeps one job per id
await queue.add('apply', JSON.parse(String(req.body)), { jobId: id });
res.sendStatus(200); // in milliseconds: the worker does the slow part
});
function verify(raw: Buffer, id: string, ts: number, header: string) {
const expected = createHmac('sha256', SECRET)
.update(`${id}.${ts}.${raw}`).digest('base64');
const sigs = header.split(' ').map((s) => s.split(',')[1]);
return sigs.some((s) => safeEqual(s, expected));
}
The worker applies the result in one transaction. Mark the webhook ID as seen; if it was already there, it's a retry, so stop. Then lock the job. If it's already finished, the event is late. That's webhook four: a job never moves backwards. If the prediction succeeded, copy the output to your own storage before it disappears, and finish the job.
The reconciler runs every five minutes and checks each job that is still running with the provider, through the same function. The webhook is a hint. The provider is the truth.
// jobs/apply.ts
const TERMINAL = ['succeeded', 'failed', 'canceled'];
export const apply = (eventId: string | null, p: Prediction) =>
db.tx(async (tx) => {
if (eventId && !(await tx.markSeen(eventId))) return; // a retry
const job = await tx.jobs.lock(p.id); // SELECT … FOR UPDATE
if (TERMINAL.includes(job.status)) return; // late: never go back
if (!TERMINAL.includes(p.status)) return; // still running
const key =
p.status === 'succeeded' ? await copyToS3(p.output, job.id) : null;
await tx.jobs.finish(job.id, p.status, key);
});
// Every 5 minutes: the webhook is a hint, the provider is the truth
export async function reconcile() {
for (const job of await db.jobs.runningFor('5 minutes')) {
await apply(null, await replicate.get(job.predictionId));
}
}
And nobody has to ask where their report is.
The numbers are targets for an illustrative scenario, not measurements.
▶️ Watch the video: https://www.youtube.com/watch?v=_Arkjh6IaZ4
💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks
🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1