{"slug": "anthropic-batch-api-112-of-my-1842-jobs-never-came-back", "title": "Anthropic Batch API: 112 of My 1,842 Jobs Never Came Back", "summary": "A developer running a mock voice interview platform found that 112 of 1,842 scoring jobs pushed through the Anthropic Batch API over 30 days silently produced no usable result — a 6.1% failure rate. The failures stemmed from treating batch.processing_status == \"ended\" as proof that every request succeeded, ignoring per-request result types (71 errored, 24 expired) and 17 responses truncated by max_tokens, plus matching results by position instead of custom_id. The developer's fix is to check batch.request_counts and each result line's result.type, and to key results by an opaque custom_id.", "body_md": "A user emailed me a screenshot of an empty report. Score field blank. I opened the database and found the row sitting there with `score = NULL`, created four hours earlier, no error logged anywhere. My worker had run, my batch had finished, my code had written nothing and moved on cheerfully.\n\nThat was the third one that week. So I pulled 30 days of logs and counted: I had pushed 1,842 scoring jobs through the Anthropic Batch API, and 112 of them never produced a usable result. Not 112 crashes. 112 silent holes, because \"the batch ended\" and \"my requests succeeded\" are two completely different facts and I had written code that assumed they were the same one.\n\n`batch.processing_status == \"ended\"` does `request_counts` and switch on each result line's `result.type` (`succeeded` / `errored` / `expired` / `canceled`).` stop_reason: \"max_tokens\"` is a truncated JSON body. My parser swallowed it and wrote a null.\nIt is asynchronous bulk inference at roughly half price. You hand Anthropic a list of up to 100,000 requests, each wrapped in a `custom_id`, and poll until the batch ends. The tradeoff is the entire point: you give up latency guarantees and you get a discount. There is a 24-hour ceiling on processing.\n\nMy workload fit on paper. I run a platform that does mock voice interviews and hands back a written report afterward, plus a separate scoring pass over the candidate's portfolio. The portfolio pass is nightly, nobody is watching, and it is a fat prompt. Perfect batch job.\n\nThe interview report is not a batch job, and I put it in one anyway. That was the first mistake and I'll get to it.\n\n```\nbatch = client.messages.batches.create(\n    requests=[\n        {\n            \"custom_id\": session_uuid,          # not the user's email\n            \"params\": {\n                \"model\": \"claude-sonnet-5\",\n                \"max_tokens\": 4096,\n                \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n            },\n        }\n        for session_uuid, prompt in pending\n    ]\n)\n```\n\nOne small thing worth doing on day one: `custom_id` comes back to you in the results file, so make it an opaque internal ID. I used the session UUID. Do not put an email address there because it was convenient.\n\nBecause I only checked one flag. Here is the real breakdown from the 30-day run:\n\n| `result.type` | count | \n|---|---|\n| succeeded | 1,747 | \n| errored | 71 | \n| expired | 24 | \n| canceled | 0 | \n\nOf the 71 errored: 52 `overloaded_error`, 14 `invalid_request_error` (transcripts that blew past my own token budget), 5 generic `api_error`. Then, of the 1,747 that succeeded, **17 came back with `stop_reason: \"max_tokens\"`** — valid API responses containing half a JSON object. My `json.loads` threw, my `except` block logged at DEBUG, and the row stayed null.\n\n71 + 24 + 17 = 112. A 6.1% silent failure rate on a pipeline I thought was green.\n\n`processing_status: \"ended\"` mean my requests succeeded?\nNo. `ended` means Anthropic is done working on the batch, including the parts it gave up on. The per-request outcome lives in two places: `batch.request_counts`, and the JSONL results stream.\n\nMy original loop, in full embarrassing honesty:\n\n```\nwhile batch.processing_status != \"ended\":\n    time.sleep(30)\n    batch = client.messages.batches.retrieve(batch.id)\n\nresults = list(client.messages.batches.results(batch.id))\nfor session, entry in zip(pending_sessions, results):   # <-- both bugs live here\n    save_report(session.id, entry.result.message.content[0].text)\n```\n\nTwo failures in one line. `entry.result.message` does not exist on an errored or expired entry, so those raise an `AttributeError` I was catching too broadly. And `zip` against `pending_sessions` assumes the results arrive in submission order.\n\nThey do not.\n\nBecause batch results come back in arbitrary order and I matched by position instead of by `custom_id`. When every request succeeds, the order is often close enough to input order that nothing looks wrong. The moment one request errors out, the list shortens by one and **every entry after it shifts up a slot**. Candidate A gets Candidate B's report.\n\nThis is the bug that actually scared me. No exception, no alert, no 500. Just quietly wrong output rendered in a nice template. I found it by reading a report that praised a Kubernetes project the person had never mentioned.\n\nFull disclosure: the product is [Preterview](https://preterview.com/en), which I built and run. It does voice interviews with three interviewer styles and returns a written report, and a report is the entire deliverable, so shipping one person's feedback to another person's account is about as bad as my failure modes get. Two sessions were affected before I caught it. I emailed both people, which is a conversation I recommend avoiding by writing the loop correctly the first time.\n\nThe fix is four lines and I should have written them on day one:\n\n```\nby_id = {}\nfor entry in client.messages.batches.results(batch.id):\n    by_id[entry.custom_id] = entry          # never trust order\n\nfor session in pending_sessions:\n    entry = by_id.get(session.id)\n    if entry is None:\n        mark_retry(session.id, \"missing_from_results\")\n        continue\n\n    match entry.result.type:\n        case \"succeeded\":\n            msg = entry.result.message\n            if msg.stop_reason == \"max_tokens\":\n                mark_retry(session.id, \"truncated\")\n            else:\n                save_report(session.id, msg.content[0].text)\n        case \"errored\":\n            mark_retry(session.id, entry.result.error.type)\n        case \"expired\" | \"canceled\":\n            mark_retry(session.id, entry.result.type)\n```\n\nEvery branch writes a row. There is no path through that block where a request disappears.\n\nMedian 9 minutes across my 96 batches. p90 was 51 minutes. p99 was 6 hours 20 minutes. And 24 requests, spread across two unlucky batches, sat until the 24-hour ceiling and came back `expired`.\n\nThat distribution is the whole design constraint. The median tempts you into thinking batch is a slightly slower sync API. It is not. You have to build for the tail, because the tail is where your users live.\n\nWhich is why the post-interview report is back on the synchronous API, where it always belonged. Someone just finished talking for 25 minutes; they are not going to refresh for six hours. The nightly portfolio rescoring stayed on batch, and that is the workload the discount was designed for.\n\nThe piece of this I'd write first next time is not the submitter. It is the reconciler: a table of `(custom_id, batch_id, status, attempts)` and a cron job that sweeps it.\n\nRules I landed on after the postmortem:\n\n`custom_id` exists in the database as `pending` before the batch is created. If it never comes back, it is already visible as a gap.`stop_reason` on success.`max_tokens` is a failure wearing a success costume.\nAfter the reconciler shipped, the next 30 days ran 2,100-ish requests with zero unresolved rows. Still had errors. Still had a handful of expiries. But every one of them now ends up retried or dead-lettered instead of vanishing into a null column.\n\n**So what happens to the jobs that never come back from the Anthropic Batch API?** Nothing happens to them. That is the problem. A batch reaching `processing_status: \"ended\"` only means Anthropic stopped working on it, and the results JSONL can contain `errored`, `expired`, or `canceled` entries alongside successes, in arbitrary order, plus `succeeded` entries truncated at `max_tokens`. If your consumer zips results against the input list and reads `.result.message` without checking `.result.type`, those requests are not reported as failures — they are dropped, or worse, shifted onto the wrong record. Match on `custom_id`, handle all four result types, verify `stop_reason`, and reconcile submitted-versus-resolved counts per batch. In my 30-day run that was the difference between 6.1% silent data loss and a clean retry queue, at half the API cost.\n\n*Written by the developer behind [Preterview](https://preterview.com/en), an interview prep platform.*", "url": "https://wpnews.pro/news/anthropic-batch-api-112-of-my-1842-jobs-never-came-back", "canonical_source": "https://dev.to/ji_ai/anthropic-batch-api-112-of-my-1842-jobs-never-came-back-2606", "published_at": "2026-09-20 05:10:21+00:00", "updated_at": "2026-09-20 05:24:20.233013+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Anthropic", "Anthropic Batch API", "claude-sonnet-5"], "alternates": {"html": "https://wpnews.pro/news/anthropic-batch-api-112-of-my-1842-jobs-never-came-back", "markdown": "https://wpnews.pro/news/anthropic-batch-api-112-of-my-1842-jobs-never-came-back.md", "text": "https://wpnews.pro/news/anthropic-batch-api-112-of-my-1842-jobs-never-came-back.txt", "jsonld": "https://wpnews.pro/news/anthropic-batch-api-112-of-my-1842-jobs-never-came-back.jsonld"}}