# The cheapest GPU is wherever people are asleep > Source: > Published: 2026-10-21 13:34:31+00:00 # The cheapest GPU is wherever people are asleep I run a product catalog site on the side. Every product that arrives goes through an AI pipeline: extract the specifications, classify it, write a description, check it against the catalog for duplicates. Nothing about it is urgent. It runs overnight, and nobody is waiting for it. For most of a year I paid interactive prices for it anyway, and this is the story of noticing. ## The bill Over several months the catalog pipeline sent about a thousand dollars to OpenAI and Anthropic. That is not a number that ruins anyone. It nagged at me for a different reason: none of the work needed to happen when it happened. A merchant feed lands at two in the morning, the enrichment runs, and the first person to look at the result is me, with coffee, hours later. I was paying for an answer in seconds and using it the next day. Interactive AI services are priced for the hard case. Somebody is typing, the model has to answer now, and the provider has to keep enough GPUs warm to answer everyone at once at the busiest minute of the day. That is expensive to run and the price reflects it. Work that can wait has none of those costs, and almost nothing in the market priced it that way. ## Two prompts, then the batch API At the start the pipeline was two prompts. One giant enrichment prompt did the extraction, the classification and the description for every platform the catalog supports, and one giant dedupe prompt judged a product against the catalog. The first fix for the bill was the obvious one: OpenAI sells a batch API, upload a file of requests, accept a 24-hour window, pay half price. I moved both prompts onto it and the bill halved. The data was already going wrong, and it kept going wrong at half price. Within a few months I had a few thousand products with generated filler in their descriptions, and duplicates the dedupe prompt had waved through, because a prompt asked to do four things does none of them carefully. Recovering from that was a project of its own. Splitting the work fixed the quality: one enrichment prompt per platform instead of one for all of them, and then steps inside each, a small extraction call, a classification that reads it, a description that reads both, a dedupe check against what the catalog already holds through my own API. It also showed what a batch API is. It takes one request at a time and has no idea that mine belong together. So the pipeline becomes: submit step one, wait up to a day, collect, write the glue that builds step two from the results, submit, wait up to a day again. Seven steps is a week in the worst case, and nothing checks a step’s output before the next step reads it unless I write that too. That glue is the part that does not get talked about. The model call is a line of code. The system around it, which holds the intermediate results, knows which step is waiting on which, retries the ones that came back malformed and refuses to pass a bad one downstream, is a small workflow engine, and every team that runs a nightly AI job ends up writing one. Mine was a few hundred lines of Go and a Postgres table, and it was the least reliable part of the site. ## A weekend in Seoul By then I was running a scheduler of my own anyway, so one weekend I did the other obvious experiment. I rented a spot GPU instance in AWS’s Seoul region, started vLLM with an open-weight model, and pointed the same steps at it. The jobs ran. The quality was fine for the task, which is extraction and classification, not poetry. And the bill for the weekend was a fraction of what the same work had cost through an API. The GPU was cheap for a boring reason. Spot capacity is whatever a cloud has not sold, offered at a discount on the condition that it can be taken back. It was the weekend in Korea. Nobody wanted that machine, so it went for very little. The same instance type on a Tuesday afternoon in Virginia would have cost several times as much, if it was available at all. That is the whole insight, and it is not a secret. The cheapest compute on earth, at any given hour, is wherever people are asleep or at the beach. Our own price index for September 2026 puts the median spot discount on an A10G at 51% off on-demand, and on an H100 at 64%. Across regions the same A10G ranged over a 10.2× spread between its cheapest and dearest region in the same month. A job whose deadline is “some time before Monday” can go and find the cheap end of that range. A job that must answer now cannot. ## The deadline is the dial Put the two halves together and the product designs itself. If the work arrives as one pipeline with one deadline, a scheduler can see the whole job: how many steps, how much input, which steps can run side by side, and how long it has. Then it can do the thing I did by hand that weekend, which is wait for cheap capacity somewhere in the world and run the job there, inside the deadline. The deadline becomes the only price lever a customer needs. An hour costs more than a day, a day costs more than a week, because a longer window gives the scheduler more hours and more regions to choose from. The customer never picks a machine or a region. They say how long they can wait, and the price follows from that. Everything else, the spot bidding, the interruptions, the model weights, the region hopping, is the platform’s problem, which is the point of a platform. The other thing a scheduler sees when it sees the whole job is the repetition. The instructions at the top of every prompt in a batch are the same ten thousand times over, and a batch that runs on one warm model can reuse that prefix instead of paying to read it again. The savings are large enough that it was simpler to make repeated input free than to meter it. ## What it became Otium is that scheduler with a workflow engine in front of it. You write a pipeline as one YAML document: each step is a prompt, a call to one of your own HTTP endpoints, or a pause for an outside system or a person. You do not declare the order; the compiler reads which step uses which result and derives the graph, then tells you before anything runs which steps are parallel, how long the job should take, and what it should cost. Each step’s output is checked against a schema and retried, or handed to a bigger model, before the next step sees it. You can unit-test the whole thing offline with canned outputs. It runs on open-weight models, which matters for a reason beyond price: a model you build a pipeline on cannot be withdrawn from under you by a vendor’s deprecation schedule. Your prompts and results are encrypted with a key that is yours, deleted on a fixed schedule, and never used for training, because the provider in this case is not in the business of training anything. It has been in production since the middle of October, with one paying customer: my catalog site, whose pipeline now runs on it instead of through an API. The parts that other people could use on their own, the spot capacity layer, the payload encryption format, the Hugging Face client, are published under Apache-2.0. The scheduler and the workflow engine are not, and I would rather say so than imply otherwise. ## If you run a pipeline like this Whether or not you use Otium, three things from this year generalise. 1. Separate the work nobody is waiting on from the work somebody is. Most teams find that the first pile is larger than they thought, and it is the pile where every pricing lever applies. 2. Make every step return a schema, and validate it before the next step reads it. Most of the glue I wrote existed to cope with a step that returned prose when it was asked for JSON. 3. Treat the deadline as a decision. “As fast as possible” is a default, not a requirement, and it is the most expensive setting on the dial. Otium is in closed alpha. If you have a nightly or weekly AI job and want to put it through, the sign-up at the foot of this page is the way in, and I read every entry myself. ## Guides this draws on [Spot, preemptible, and remnant GPU compute](https://getotium.ai/learn/spot-and-remnant-compute) Spot and remnant GPUs are the same hardware sold cheap, with the catch that it can be reclaimed. How it works, and why batch can use it when interactive cannot. [Migrating from the OpenAI Batch API](https://getotium.ai/learn/openai-batch-api-migration) OpenAI-compatible batch surfaces make switching providers a base-URL change, not a rewrite. How the file-and-batch flow works, and what to check when you move. [From chained calls to one pipeline: designing multi-step AI work](https://getotium.ai/learn/multi-step-ai-pipelines) Most useful AI work is several model calls in a row. The common patterns, where chained jobs lose time and money, and how to design steps that hold up. [Workflow-aware inference: why the request is the wrong unit](https://getotium.ai/learn/workflow-aware-inference) If your AI work is a pipeline, submitting one request at a time discards the structure that would let it be scheduled well. What the research says that costs. [How to choose a deadline for AI work](https://getotium.ai/learn/choosing-a-deadline) Work backwards from when the output is needed, not from how fast it could be. A practical way to set deadlines for batch AI jobs and pipelines. [What an LLM inference request really costs](https://getotium.ai/learn/llm-inference-cost) GPU compute cost vs. total serving cost: how to measure throughput, what moves it, and a worked example with every assumption stated.