{"slug": "show-hn-submit-asynchronous-llm-batch-jobs-through-one-interface", "title": "Show HN: Submit asynchronous LLM batch jobs through one interface", "summary": "A developer released batchlane, an open-source Python library that submits asynchronous LLM batch jobs through a single interface spanning eight providers: Anthropic, Gemini AI Studio, OpenAI, Groq, Mistral, Fireworks, Together, and DeepInfra. Only the Anthropic adapter has been verified end to end against a live API, while the other seven adapters have mocked contract tests. The library handles chunking, polling, and rejoining results to input rows, supports resumable checkpoints that store job handles rather than answers, and refuses requests where no provider lane exists.", "body_md": "Submit asynchronous LLM batch jobs through one interface.\n\nEight adapters cover Anthropic, Gemini AI Studio, OpenAI, Groq, Mistral, Fireworks, Together, and DeepInfra. Batch pricing, model availability, and turnaround depend on the provider. Only Anthropic has been verified end to end against a live API; the other adapters have mocked contract tests.\n\n``` python\nimport batchlane as bl\n\nmodel = \"groq/llama-3.3-70b-versatile\"\nprompts = [\"The product was great.\", \"It broke in a week.\"]\nanswers = bl.map(model, prompts, system=\"Classify the sentiment.\")\n```\n\nOne prompt over many inputs, answers back in input order, `None` where a row\nfailed. Underneath it splits the job to fit the provider's caps, submits\nhowever many batches that takes, waits, and rejoins the results.\n\nFrom the shell, for the file-to-file case:\n\n```\nbatchlane run rows.jsonl --model groq/llama-3.3-70b-versatile -o answers.jsonl\n```\n\nEvery field on an input row is copied to its output row beside a new `answer`,\nso your own columns stay attached to their results. `--dry-run` reports the\nchunking without submitting anything; `batchlane providers` lists the lanes.\n\nWhen you need per-row control, `run()` gives you each input back beside its\nresult:\n\n```\nrows = [\n    bl.BatchLine(\n        \"row-1\",\n        \"groq/llama-3.3-70b-versatile\",\n        [{\"role\": \"user\", \"content\": \"Classify: the product was great\"}],\n    ),\n    bl.BatchLine(\n        \"row-2\",\n        \"groq/llama-3.3-70b-versatile\",\n        [{\"role\": \"user\", \"content\": \"Classify: it broke in a week\"}],\n    ),\n]\n\nfor line, result in bl.run(rows, checkpoint=\"job.jsonl\"):\n    print(line.custom_id, bl.answer_text(result))\n```\n\nResults are yielded one row at a time. Requests and downloaded provider output are currently buffered in memory; size jobs to fit your machine.\n\nWhat that buys you:\n\n| **Provider batch pricing** | Discounts and model exclusions vary; see the provider table below | \n| **One code path** | chunking, polling, and joining results back to rows are handled | \n| **Resumable** | recorded handles reattach to submitted jobs; recovery limits are described below | \n| **Honest about limits** | refuses where no lane exists rather than emulating one, and distinguishes \"no lane\" from \"not built yet\" | \n\nWith `checkpoint=` set, batchlane records an intent before each submission and\nsaves the returned handle immediately. Repeating the identical call reattaches\nto recorded jobs. Changed requests, model parameters, ordering, or job settings\nare rejected; use a new checkpoint for new work. Use one writer per checkpoint.\n\nIf a submission may have succeeded but no handle was saved, recovery depends on the provider's listing and matching support. It is not an exactly-once guarantee. Providers retain results for a limited time, so save collected answers locally if you need them beyond that window. Checkpoints store handles, not answers.\n\n`run()` submits every chunk before waiting. To submit now and collect later:\n\n```\nhandles = bl.submit_all(rows, checkpoint=\"job.jsonl\")\nfor handle in handles:\n    if bl.status(handle).state == \"succeeded\":\n        for result in bl.results(handle):\n            print(result.custom_id, bl.answer_text(result))\n```\n\n`plan(rows).chunks` describes which requests each handle covers.\n\nUse native Responses `input` with `endpoint=\"responses\"`. Leave `messages`\nempty and put other Responses parameters in `params`:\n\n``` python\nimport batchlane as bl\n\nlines = [\n    bl.BatchLine(\n        \"page-1\",\n        \"openai/gpt-4o-mini\",\n        input=[\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\"type\": \"input_text\", \"text\": \"Summarize this page.\"},\n                ],\n            }\n        ],\n        params={\"max_output_tokens\": 200},\n    )\n]\nhandles = bl.submit_all(lines, endpoint=\"responses\", checkpoint=\"responses.jsonl\")\nfor line, result in bl.run(lines, endpoint=\"responses\", checkpoint=\"responses.jsonl\"):\n    print(line.custom_id, bl.answer_text(result))\n```\n\nInputs can include native image and file blocks. Result bodies retain native\noutput items, refusals, tool calls, reasoning details, and usage; `answer_text`\nextracts only output text. This uses the [OpenAI Batch API](https://developers.openai.com/api/reference/resources/batches/methods/create).\nStreaming and background mode are not batch request modes. Other provider lanes\ncontinue to accept chat-completion requests. Responses input tokens and costs\nare not estimated offline; `actual_cost` can price collected usage.\n\n```\nfor note in bl.plan(rows).caveats:\n    print(note)\n```\n\nLanes differ in ways that change what you should do, not just how the client\ntalks to them. `plan()` states the ones that apply to your job, and the CLI\nprints them on every run. `batchlane run ... --dry-run` shows them with the\ncost and the chunking and submits nothing.\n\n```\nresults = list(bl.results(handle))\nprint(bl.actual_cost(results, handle.provider))\n```\n\n`plan().cost` estimates a job before it runs; `actual_cost()` prices usage\nreported by the provider. Where a provider returns a service tier, the cost\nreport carries a caveat if that tier differs from batch pricing.\n\n```\np = bl.plan(rows)\nprint(p.n_chunks, p.total_bytes, p.limit_bytes)\nprint(p.cost)\n```\n\nThe `~` marks a derived rate. Where batch rates are absent from LiteLLM's price\nregistry, batchlane applies the provider discount in its capability table.\nTogether has no estimate because its discount varies by model. Output length\nis unknown before inference; `max_tokens` bounds the output estimate, while its\nabsence limits the estimate to input tokens. These estimates are not quotes\nor spending limits.\n\nGemini uses inline requests below 20MB and switches to keyed JSONL file input\nfor larger batches, with a 2GB provider file limit. `plan()` includes request\nand byte limits when splitting work. You can request smaller chunks with\n`plan(rows, max_requests_per_batch=1000)` and use the same argument on\n`submit_all()`.\n\nFor a single batch:\n\n``` php\nhandle = bl.submit(rows)  # -> BatchHandle, JSON-serializable\nopen(\"job.json\", \"w\").write(handle.to_json())\nbl.wait(handle)  # poll until terminal\nlist(bl.results(handle))  # joined on your custom_id\nbl.cancel(handle)\nbl.list_jobs(\"groq\")\npip install 'batchlane[serve]'\nbatchlane serve\npython\nimport openai\n\nclient = openai.OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"unused\")\n\nf = client.files.create(file=open(\"rows.jsonl\", \"rb\"), purpose=\"batch\")\nbatch = client.batches.create(\n    input_file_id=f.id, endpoint=\"/v1/chat/completions\", completion_window=\"24h\"\n)\n```\n\nNothing above is batchlane-specific. It is the stock OpenAI SDK, and the rows\nname `groq/...` or `gemini/...` models, so the batch runs on a provider the\nclient has never heard of. An R, JavaScript or curl client works the same way:\nchange `base_url` and nothing else. The test suite proves this by driving the\nreal `openai` package against the app rather than a client of our own.\n\n**The gateway stores no jobs.** OpenAI's protocol is already state-passing at\nthe client boundary, so the `batch_id` carries the compressed handles rather\nthan pointing at a row: one process, no database, no migrations, nothing lost\non restart. Uploaded files do go to disk, because a file must survive until a\nbatch references it, and sufficiently large handle collections spill to disk when their encoded\nIDs exceed the gateway limit. `GET /v1/batches` returns an\nempty list, since a server holding nothing has nothing to enumerate.\n\n**Running it exposes spending authority.** The gateway submits jobs with your\nprovider keys, so anyone who can reach it can spend your money. It binds\nloopback by default and **refuses to serve on any other address without a\nkey**:\n\n```\nbatchlane serve --host 0.0.0.0 --api-key \"$(openssl rand -hex 16)\"\n```\n\nClients then send that as their `api_key`. Provider credentials stay in the\ngateway's environment and never reach a client.\n\nNote also that a `batch_id` is a bearer capability: it carries the job, so\nwhoever holds it can poll and read that job's results. That is what makes the\nserver stateless, and it is the trade being made.\n\nA solo user never has to run any of this; the library alone is enough.\n\nBatchlane does not send concurrent synchronous requests as a substitute for a\nprovider batch API. It raises `NoBatchLaneError` for providers classified as\nhaving no lane and `AdapterNotShippedError` where an adapter is missing.\n\nA provider whose lane exists but is unimplemented gets a *different* error, so\na refusal never claims a lane is absent when it is merely unwritten.\n\n`plan().cost` estimates cost and `actual_cost()` prices reported usage. Neither\nblocks submissions when a budget is exceeded.\n\n| Provider | Discount | Window | Live-verified | Notes | \n|---|---|---|---|---|\n| Anthropic | 50% | none | **yes** | inline requests, no file upload | \n| Gemini AI Studio | 50% | none | pending | inline or file input; see joining limits below | \n| OpenAI | 50% | 24h | no | the reference lane; litellm covers it too | \n| Groq | 50% | 24h or 7d | no | model allowlist | \n| Mistral | 50% | any Nh | no | model scoped to the job, not the line | \n| Fireworks | 50% | 12h to 72h | no | dataset upload; no cancel endpoint | \n| Together | up to 50% | 24h fixed | no | some models excluded from batch | \n| DeepInfra | 20% | 24h | no | model must be uniform across the file | \n\n\"Live-verified\" means a real batch was submitted, polled and read back, with answers checked against their inputs. Take the others as untested: their wire shapes have been checked line by line against each provider's own API reference, which is not the same as evidence that they work. The adapter module docstrings cite the source for each.\n\nGemini carries a hazard worth stating plainly: its docs say inline results map\nto requests **by array index**, not by the key you supply. batchlane joins on\nan echoed key wherever the payload carries one, falls back to submission order\notherwise, and refuses outright when the counts disagree, because a quietly\nmis-joined batch attaches plausible answers to the wrong rows and nothing\nabout the output looks wrong.\n\nxAI is skipped on purpose: its lane discounts 20% rather than 50%, and its own docs exclude the flagship models.\n\nAzure, Vertex AI and Bedrock are unshipped because litellm already reaches them, so batchlane points you there instead of claiming they have no lane.\n\nGroq, Together, DeepInfra, and Fireworks are reachable through the same interface.\n\nSelf-hosted runtimes get a different answer again. Ollama, LM Studio,\nllamafile and vLLM have no batch lane because there is no per-token price to\ndiscount: the hardware is yours already. What helps there is throughput, not a\ndiscount, so batchlane says so. For vLLM it names the command that does the\njob, `vllm run-batch`, which litellm's own hosted_vllm support will not do\nbecause it assumes an HTTP `/v1/batches` that a stock `vllm serve` does not\nexpose.\n\n```\n>>> bl.capabilities_for(\"groq\").window.allowed\n('24h', '7d')\n>>> bl.capabilities_for(\"groq\").result_retention\ndatetime.timedelta(days=30)\n```\n\nThe descriptor carries the asymmetries that quietly cost you a run: result retention (Gemini keeps results 6 weeks, Groq 30 days), whether cancel exists at all (Fireworks has no cancel endpoint), whether the window is yours to set, and which endpoints the lane covers.\n\nUse the interface that supports the provider and account you need. Batchlane\nprovides its own provider adapters, request planning, and resumable submissions.\nIt uses LiteLLM for request and response conversion; see the\n[LiteLLM batch documentation](https://docs.litellm.ai/docs/batches) for its current\nprovider support.\n\n`batchlane` depends on LiteLLM the *library* and ignores LiteLLM the gateway.\nOne module, `translate.py`, imports it, and calls nothing but pure synchronous\ntransforms. A golden-output test pins their results so a version bump fails in\nCI rather than corrupting a 50,000-row job.\n\n```\npip install batchlane\n```\n\nRequires Python 3.11 or newer.\n\nCredentials come from the usual environment variables (`ANTHROPIC_API_KEY`,\n`GEMINI_API_KEY`, `OPENAI_API_KEY`, `GROQ_API_KEY`, `TOGETHER_API_KEY`,\n`DEEPINFRA_TOKEN`), or pass `api_key=` explicitly.\n\nBatch APIs are generally excluded from free tiers: Groq's needs the Developer plan and Gemini's needs the paid tier. No provider offers a Stripe-style test key, because inference costs real compute whoever is asking.\n\n```\nuv sync --all-groups\nuv run pytest              # unit + contract, no network\nuv run ruff check .\nBATCHLANE_LIVE=1 uv run pytest -m live    # real API calls, real (tiny) spend\n```\n\nMIT", "url": "https://wpnews.pro/news/show-hn-submit-asynchronous-llm-batch-jobs-through-one-interface", "canonical_source": "https://github.com/gojiplus/batchlane", "published_at": "2026-09-23 23:33:56+00:00", "updated_at": "2026-09-24 00:01:17.869331+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["batchlane", "Anthropic", "Gemini AI Studio", "OpenAI", "Groq", "Mistral", "Fireworks", "Together"], "alternates": {"html": "https://wpnews.pro/news/show-hn-submit-asynchronous-llm-batch-jobs-through-one-interface", "markdown": "https://wpnews.pro/news/show-hn-submit-asynchronous-llm-batch-jobs-through-one-interface.md", "text": "https://wpnews.pro/news/show-hn-submit-asynchronous-llm-batch-jobs-through-one-interface.txt", "jsonld": "https://wpnews.pro/news/show-hn-submit-asynchronous-llm-batch-jobs-through-one-interface.jsonld"}}