The pitch for the OpenAI Batch API is simple. You have a pile of work nobody needs answered in two seconds, say 40,000 rows to classify overnight. You put the requests in a JSONL file, upload it, and the provider finishes within 24 hours at half the price of the normal API. The discount is real. Wha
t eats it is the code around the batch. A batch job lives for hours, and anything that lives for hours meets crashes, retries and deadlines. Each of those can quietly turn "half price" into "full price, twice", or into results that are silently missing. Here are six mistakes I made or nearly made while building a batch pipeline in .NET 10 (official OpenAI 2.14.0 package, SQLite for state).
The steps are general. The code is trimmed to the shape, and the trimmed parts are marked. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.
The complete, working, tested version of this batch project is inside, with full source and an in-depth PDF guide. Join Tech Skill Builder This is the expensive one. Your worker uploads the file, calls "create batch", and dies before it saves the batch id. On restart, your database says this shard was never sent, so you send it again.
Now two batches run the same requests, and you pay for both. The fix doesn't need extra infrastructure. Batches accept metadata, so tag each one with your own shard id when you create it. On restart, before re-sending a shard that has no batch id, list the recent batches and look for that tag: var metadata = new Dictionary<string, string> { ["job_id"] = job.Id, ["shard_id"] = shard.Id, }; // create the batch with this metadata, then save the batch id (full flow in the complete project) // on restart, for a shard with no saved batch id: var match = recent.FirstOrDefault(b => b.Metadata.TryGetValue("shard_id", out var sid) && sid == shard.Id); // match found: adopt it.
none found: send it. Never both. The other half of the rule: save your state before every provider call (planned, then uploaded, then created), so a restart always knows how far it got. A batch comes back with failures, and the easy move is "put the failures in the next batch".
Some of those failures will fail again no matter how often you send them. A 400 for a malformed request or a model refusal gets the same answer the second time, and you pay for it again. So sort the failures before you retry anything: Retry: 408, 409, 429 and 5xx, batch_expired, lines that never came back, and output that parsed but broke your schema. Retry with more room: finish_reason: length.
The next attempt doubles max_completion_tokens, up to a cap. Don't retry: other 4xx, refusals and content-filter stops. And give every item an attempt budget (three in my project). When it runs out, the item goes to a dead-letter state with its last reason instead of looping forever.
One more thing on cost: when an attempt is truncated, you still paid for its tokens. My store adds each attempt's tokens to the item rather than replacing them, so the cost report shows what you actually paid. The docs say it plainly: output lines may not come back in input order. That part most people catch, and they join on custom_id.
The subtler bug is which side you loop over. If you loop over the output file, items with no line at all simply never show up. Cancelled batches, expired batches and the occasional lost line all look like that. The job reports "done" and a few hundred rows were never processed.
Loop over what you sent instead, and look each one up: foreach (var (itemId, attempt) in shard.Members) { if (decisions.ContainsKey(itemId)) continue; // a line came back for it // no line at all: retry, with a reason like "missing_after_cancel" or "missing_after_expiry" // ... (reason mapping and the rest of the reconciler are in the complete project) } Also read the second file. Failed requests go to a separate error file, and that's where expired requests show up. Item r-042 fails in batch one and goes into batch two.
If its custom_id is still r-042, a late line from batch one is indistinguishable from the fresh answer, and it can overwrite a newer result. Put the attempt number in the id. My format is {jobId}:{itemId}:a{attempt}, so the retry is ...:r-042:a2. Any line carrying a1 after that is stale, and the reconciler ignores it. Lines from another job are ignored the same way, and a second line for the same id counts as a duplicate. The database side gets the same treatment: an update only lands if the item is still at that attempt and still waiting for a result, so a late line can't overwrite newer state. Each input file has limits: 50,000 requests and 200 MB. The default System.Text.Json encoder escapes every non-ASCII character as \uXXXX.
For English text, that changes nothing. For Arabic it hurts. In my measurement, one short Arabic ticket went from 116 bytes to 324. On long non-English rows, the byte limit starts to bind before the 50,000-request limit, so the same job needs more files and more batches. One option fixes it: var options = new JsonWriterOptions { Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }; // write custom_id, method, url, body with a Utf8JsonWriter using these options (builder omitted) Despite the scary name, it still escapes control characters, so a newline inside a row can't split a JSONL line. I have a test for exactly that, because it's the first thing I'd worry about. While you're there, validate the file locally before up: unique custom_ids, one model per file, url equal to the batch endpoint, no stream: true, and within the limits. A mistake caught locally costs nothing.
The Batch API promises 24 hours. Your report is due at 08:00. At 07:40 the batch is still running and half the rows are done. The panic move is "cancel it and run everything through the normal API".
That pays full price for the rows the batch already finished, and a cancelled batch still produces output for what it completed, which you're billed for too. The better question is when to stop trusting the batch, and how to finish only what's left without paying twice. My project answers that with a deadline planner. How it decides, and what it costs, is in the complete project and its PDF guide, not here.
[ ] State saved before every provider call [ ] Every batch tagged with your shard id; crash recovery adopts before it re-sends [ ] Failures classified: retry, retry with more tokens, or fail for good [ ] Attempt budget and a dead-letter state [ ] Tokens added across attempts, not replaced [ ] Reconcile over what you sent, using both the output file and the error file [ ] Attempt number inside custom_id; stale and foreign lines ignored [ ] Relaxed JSON encoder, and a local file validator [ ] A deadline plan that doesn't throw away finished work The project has 162 passing tests (146 unit, 16 integration), with no API key needed. An offline simulator covers the documented batch lifecycle, including failed, expired and cancelled batches, and a test crashes the process right after "create batch" to prove recovery adopts the existing batch instead of creating a second one. What I didn't do: run it against the live Batch API. The OpenAI adapter was tested against an in-process server that speaks the documented Files and Batches format, not against OpenAI itself.
Also note that BatchClient is marked experimental (OPENAI001) in the 2.14.0 SDK, so expect it to change. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. This batch project is already inside: the full .NET 10 solution with the 162 tests, the crash recovery, the reconciler, the deadline planner, a minimal API with idempotent job creation, and a PDF guide that goes much deeper than this post.
You also get the other complete, tested .NET + AI projects in the membership, with new ones added regularly. Join Tech Skill Builder for $5/month Which of these six would hit your batch job first?
Key Takeaways #
- •A batch job lives for hours, and anything that lives for hours meets crashes, retries and deadlines.
- •This story was reported by Dev.to , covering developments in thedev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article: