In 1946, the 30-ton ENIAC filled a gymnasium-sized room. Fifty years later, Penn students reproduced its architecture and capacity on a fingertip-sized chip. Computing’s familiar story sent yesterday’s machine from room to desk to pocket. Penn’s ENIAC retrospective
But engineers spent density in another direction too. ASCI Red kept the room and crossed one teraFLOP. Internet companies filled warehouses with coordinated servers. By 2009, Google engineers called the data center a “warehouse-scale computer.” Density could put yesterday’s workload in less space, or more workload inside yesterday’s walls. DOE on ASCI Red Google on the warehouse-scale computer
Give several AI endpoints the same hard question under a ten-second deadline. One spends nearly the whole interval on its answer, while another responds almost at once, and we call it “blazingly fast.” That language conflates speed with time. We describe latency in the language of speed, but what impressed us was the short wait, not the rate. Decode speed measures output tokens per second, while latency measures time to a useful result. Faster decoding creates a choice: finish the same computation sooner or put more computation inside the deadline. Call that useful extra work the deadline dividend. It can fund another strategy, a critic, a check, or recovery after failure.
On August 13, the thought experiment became a product. OpenAI opened a limited API preview of Ultrafast for GPT-5.6 Sol and reported up to 750 output tokens per second, as much as 14 times Standard speed. In Cerebras’s six-task GDP-Val comparison at medium reasoning, GPT-5.6 Sol finished in 83.0 seconds on Ultrafast and 464.2 seconds on Standard, a 5.59-times end-to-end gain under settings Cerebras describes as quality-matched. The faster path runs on Cerebras. OpenAI on Ultrafast Cerebras benchmark
Artificial Analysis’s August 13 snapshot supplies a wider comparison for gpt-oss-120b. Cerebras decoded at 1,790.3 tokens per second. SambaNova, the next-fastest listed endpoint, reached 701.3 tokens per second, compared with 476.1 for Groq and 170.7 for the 16-endpoint median. Cerebras ran 2.55 times as fast as the runner-up and 10.49 times as fast as the median. Live provider comparison Captured endpoint vector
The snapshot also scored the configured Cerebras endpoint at 87.28% accuracy and SambaNova at 98.15%. For capability, Sol provides the stronger comparison because GPT-5.6 Sol ran at medium reasoning on both paths under settings Cerebras describes as quality-matched.
Assume one second of overhead, a 500-token answer, and the measured rate for sequential generation. Generated-token headroom reaches 1,036 at the provider median, 5,812 for the runner-up, and 15,613 for Cerebras. For gpt-oss’s exposed reasoning trace, Cerebras offers a modeled token budget 2.69 times the runner-up’s and 15.07 times the median’s. How much more capability can that buy? Existing reasoning and selection methods can spend the extra serial tokens now. Future models can learn to use larger TTC budgets well. Faster inference can also put model decisions on the live execution path, where serial agent loops compound the gain.
What the speed test measures #
The gpt-oss comparison uses 10,000-token prompts and rolling 72-hour medians. Each endpoint combines hardware, quantization, software, scheduling, and provider settings. The capture recorded Cerebras’s first chunk at 0.53 seconds and first answer token at 1.65 seconds.
Time to first token includes request handling, queueing, prefill, and initial generation. Once text starts, output speed records the average number of tokens received per second. Its reciprocal is average inter-token latency. End-to-end latency measures the full wait for a useful result. Batching can raise fleet throughput independently of the stream speed seen by one user.
Artificial Analysis’s benchmark sends one prompt at a time and calculates output speed after the first token. gpt-oss exposes an autoregressive reasoning trace, so the measured rate shows how much fits before a deadline. Performance methodology OpenAI on gpt-oss
Spending the dividend #
Researchers call computation spent after a prompt test-time compute, or TTC. Intrinsic TTC can lengthen one autoregressive path, whose pace depends on single-stream decode. Extrinsic TTC adds samples, critics, verifiers, or tools around the model. Parallel branches trade concurrency and aggregate compute for less wall-clock time. A selector must still turn candidate coverage into one answer.
Training determines whether extra inference compute helps; the request budget determines how much of the learned process runs. Current systems can spend the dividend on another strategy, a contradiction check, verification, or recovery after failure.
OpenAI’s o-series paired larger TTC budgets with a generational jump in reasoning performance. OpenAI says reinforcement learning taught o1 to reason productively, with performance rising alongside training compute and thinking time. On AIME 2024, single-sample accuracy rose from GPT-4o’s 12% to o1’s 74% at its highest TTC setting. For o3, OpenAI reported another order of magnitude in reinforcement-learning compute and inference-time reasoning, with further gains. Snell and colleagues made prompt-adaptive TTC over four times as efficient as best-of-N on tested math problems. In some FLOPs-matched cases, a smaller PaLM 2 beat one roughly 14 times larger. OpenAI on o1 OpenAI on o3 Snell et al.
Cerebras packages this approach as CePO: plan, sample, compare inconsistencies, then choose through structured Best-of-N. Cerebras reports spending 10 to 20 times the one-shot budget, generating about 100 tokens per second on Llama 3.3-70B, and beating Llama 3.1-405B on selected benchmarks. OptiLLM can send the workflow to several providers. CePO shows one concrete way to spend inference speed on additional reasoning and selection. CePO Documentation OptiLLM
When inference becomes execution #
Karpathy calls natural-language programs executed by an LLM “Software 3.0.” When a model decides what a live application does next, inference latency becomes execution latency. One decision can create value. A fraud signal before a transaction clears can block it, while an incident diagnosis during a failure can guide the response. Faster frontier inference can move those decisions from a background process onto the live path. Tool use compounds the effect because the model emits a call, the client executes it, and the result becomes the next state. Cost determines how many calls a product can afford; latency determines how many it can put in a row. Routing can send routine steps to a smaller model and hard turns to a frontier model. “Software Is Changing (Again)” Tool calling RouteLLM
Codex-Spark shows what happens when model generation stops dominating the clock. OpenAI runs the smaller real-time coding model on Cerebras above 1,000 tokens per second. API overhead became material, prompting caching, fewer network hops, and persistent WebSockets. OpenAI reports agent loops up to 40% faster end to end. Developers can interrupt, redirect, edit, test, and retry midstream. Codex-Spark OpenAI on agent loops
plan → search → inspect → edit → test → diagnose → retest → report
Give each step 0.75 seconds of fixed overhead, 1,000 output tokens, and a 1.5-second tool round trip. At the 170.7-token-per-second provider median, the loop takes 64.9 seconds. At Cerebras’s captured 1,790.3-token rate, it takes 22.5 seconds. Under those assumptions, a 30-second deadline requires at least 667 tokens per second. At that threshold, speed decides whether the program runs in the request path or a background job.
Cerebras’s public API currently lists two tool-capable models. Its dedicated catalog lists 43 configurations from 12 publishers, plus custom weights. Enterprises can route routine turns to smaller models and hard turns to larger ones. Public model API Dedicated model catalog
Exhibit: when raw speed survives the clock #
Under the stated assumptions, the base ten-second case yields 2.69 times the runner-up’s modeled serial-token budget and 15.07 times the provider median’s. Show the model and sensitivity table
A fixed deadline has to accommodate overhead, tools, verification, and the answer. A completion cap can further constrain the reasoning budget.
For one serial path, model total time as
T
, F
, B
, A
, s
, U
, and V
represent the deadline, fixed request overhead, reasoning tokens, answer tokens, decode rate, tool time, and verification or selection, respectively. Fixed overhead includes the queue, prefill, and time to first token. With completion cap C
, B = max(0, min{s(T-F-U-V)-A, C-A})
. Fitting m
times the baseline budget requires
Parallel branches can overlap in wall time, but each adds total compute. The table sets U
and V
to zero to isolate serial work. The loop above includes tool time. Rates come from the capture. Deadlines, overhead, answer lengths, the four-times target, and constant rates are assumptions.
| Sensitivity | Assumptions T / F / A |
Baseline to fast rate | Baseline to fast B |
B multiple |
Required / observed speedup |
|---|---|---|---|---|---|
| Low tolerance, next-fastest | 2s / 0.5s / 100 | 701.3 to 1,790.3 | 952 to 2,585 | 2.72 times | 3.71 / 2.55 times |
| Base, next-fastest | 10s / 1s / 500 | 701.3 to 1,790.3 | 5,812 to 15,613 | 2.69 times | 3.76 / 2.55 times |
| Base, provider median | 10s / 1s / 500 | 170.7 to 1,790.3 | 1,036 to 15,613 | 15.07 times | 3.02 / 10.49 times |
| High tolerance, capped | 60s / 1s / 1,000 | 170.7 to 1,790.3 | 9,070 to 39,960 | 4.41 times | 3.70 / 10.49 times |
At ten seconds, modeled token headroom grows 2.69 times versus the runner-up and 15.07 times versus the median. At 60 seconds, Cerebras’s 40,960-token completion cap leaves 39,960 tokens before the assumed answer. Cerebras model metadata
The next scaling lever #
Current models can spend that room on longer thought, another candidate, a check, or another action. Training models to use larger budgets well could add a scaling lever.
Training scaling followed a simple recipe: add parameters, data, and compute. DeepMind found model size and training tokens should grow together, increasing data demand. Epoch AI estimates frontier developers could use the quality- and repetition-adjusted stock of public human text between 2026 and 2032 if trends hold. Synthetic, private, multimodal, and reused data could extend the runway. OpenAI’s scaling-law paper Chinchilla Epoch AI on public human text Scaling data-constrained models Epoch AI’s 2030 synthesis
Pretraining commits compute and data before deployment. TTC allocates compute after a request, giving harder problems larger budgets. Reasoning-oriented training can make inference compute more productive. Faster infrastructure can make larger budgets practical within a fixed deadline. Future training can then target that expanded budget.
From a fast endpoint to a market position #
Speed becomes a large business when answers lose value while the model works. OpenAI highlights incident response, finance, voice, commerce, and research. Crossing from a background job to a live interaction enables a different product. Its commercial value depends on the all-in cost of a correct, timely outcome. OpenAI on Ultrafast
Cerebras credits the WSE-3’s 900,000 cores, 44 GB of SRAM, and 21 PB/s of memory bandwidth for its performance. It pipelines model layers across wafers, keeping weights close to compute. Artificial Analysis independently measures the complete endpoint. OpenAI’s launch puts its flagship model on Cerebras. Prospectus Ultrafast architecture
OpenAI committed to buy 750 MW through 2028, advanced about $1.0 billion, and holds an option for another 1.25 GW by 2030. It already runs Codex-Spark and Sol Ultrafast on Cerebras, putting its flagship model on wafer-scale inference through OpenAI’s API. Q2 10-Q OpenAI partnership Codex-Spark Ultrafast
Cerebras says it agreed with OpenAI to co-design future models for future Cerebras hardware, a path for training and hardware to adapt to a shared latency envelope. SEC prospectus
Amazon plans Trainium for parallel prefill and Cerebras CS-3 for serial decode. Its agreement includes a multiyear lease and purchase options, and Cerebras targets production in Q1 2027. That split defines a valuable wedge because Cerebras can specialize in serial decode while other hardware handles training, prefill, or batched serving. AWS Q2 release
At June 30, Cerebras held $8.6 billion in cash, cash equivalents, restricted cash, and short-term investments. It says more than 600 MW was live or contracted for delivery by year-end 2027, 2026 manufacturing capacity would grow more than tenfold, and required TSMC wafers were secured. Q2 release
Artificial Analysis lists the measured Cerebras endpoint at $0.75 per million output tokens, 25% above the provider median. In the base case, quadrupling the modeled serial budget uses 3.02 times as many output tokens and 3.78 times the spend. The larger budget earns its premium when verification, recovery, or better answers lower the all-in cost of a correct, timely result. Provider pricing
Competitors attack other parts of the clock. Batching raises throughput, while smaller models, caching, and routing reduce work. Faster tools and pipelines cut external and first-token delays. For supported models on which Cerebras sustains a matched-quality decode lead, it accelerates serial generation directly. Provider comparison Codex-Spark
The OpenAI contract has begun turning into revenue. Cerebras recorded $180.1 million of GAAP Q2 revenue, including $126.0 million from cloud and other services and $56.8 million under the OpenAI arrangement. Remaining performance obligations reached $25.4 billion. Core cloud gross margin rose from 26.0% to 41.8%. GAAP cloud gross margin was 19.5% after customer-warrant amortization, stock-based compensation, and pass-through revenue and costs. 10-Q Q2 release
Cerebras sits at the junction of all three. CePO turns fast inference into additional reasoning and selection today. The OpenAI co-design agreement opens a path toward models trained for larger TTC budgets. Cerebras’s tool-capable models and broader catalog give Software 3.0 a fast execution layer across model sizes. OpenAI distributes Spark and Sol, while AWS adds a future cloud channel. Catalog CePO Ultrafast OpenAI agreement AWS
As components grew denser, engineers could put the same computation in a smaller machine or keep the building and put more computation inside it. Fast inference recreates that choice in time. The same speedup can finish the old workload sooner or keep the deadline and fill it with more reasoning and action.
By the end of 2028, Cerebras must beat the best matched-quality commercial alternative by at least twofold at p95 end-to-end latency under the same output contract and disclosed concurrency. Decode must account for the largest reducible delay, added TTC must lower all-in cost per correct result, and paid service must produce a positive contribution margin. Clear those hurdles on one serial workload, and Cerebras has a wedge. Repeat the result, and the company sells something much larger than speed: more capability and more action before the clock runs out.
Linked source notes #
Evidence key.Fact marks directly reported data within its stated scope. Interested-party claim flags a vendor, customer, or issuer with commercial incentives. Calculation follows displayed assumptions. Inference interprets evidence. Speculation names a testable possibility. Sources were accessed August 11 through 18, 2026. Artificial Analysis figures use the August 13 capture.
Historical sources:PennDOEGoogle ResearchBenchmark scope: Artificial Analysis measures single requests and rolling medians. The capture preserves 16 raw rates, the calculated 170.7 median, endpoint-accuracy scores, and configuration details as the live window rolls.Live tableCaptureMethodResearch scope: Additional or adaptive TTC improved the cited models on the cited tasks, budgets, and verifiers. Serial reasoning, sampling, and selection have distinct scaling curves.o1o3SnellBrownOpenAI and Chinchilla cover training scaling; Epoch projects public-text scarcity but no overall 2030 data bind; Muennighoff et al. test reuse.OpenAI scalingChinchillaEpoch dataData-constrained scalingEpoch 2030Commercial and contract scope: OpenAI reports Ultrafast at up to 750 output tokens per second and 14 times Standard; Cerebras reports the quality-matched six-task result. Price, loaded tail latency, concurrency, and energy complete the commercial test. Three customers accounted for 76% of Q2 revenue. OpenAI’s warrant covers up to 33,445,026 shares at $0.00001; 10,033,508 were exercised in July. The 750 MW arrives through 2028. Specified delivery delays or service failures can let OpenAI terminate part or all of the agreement; such termination, except for OpenAI’s uncured material breach, may accelerate the loan. Cerebras expects 22% of the $25.4 billion obligation through June 2028, 43% in months 25 through 48, and the balance later.OpenAICerebras10-QFinancial scope: Cerebras guided to roughly $194 million of Q2 core revenue and reported $209.9 million; it reported $180.1 million under GAAP, below the $194 million LSEG consensus cited by CNBC. The company then raised full-year core revenue guidance to $880 million to $890 million. Customer-warrant amortization reduced Q2 reported revenue by $44.3 million, tying part of shareholder dilution to commercial milestones. Against $8.6 billion of reported liquidity, Cerebras had $1.5 billion of uncommenced data-center lease payments at quarter-end and contracted another $753.4 million afterward.Q1 guidanceQ2 release10-QCNBC/LSEG comparisonReproducibility: Calculations use the full-precision captured rates; printed values are rounded. The eight-step reliability sensitivity assumes independent 95% per-step success, yielding 66.3% full-loop success; faster cycles create room for retries. TokenArena supplies a modeled service-energy framework.Captured endpoint vectorModel metadataTokenArena