You’re evaluating local inference for your developers. You’d like more control over your code and your AI bill, but you need to know what happens when the whole team starts using it. Will responses arrive quickly enough? Can the model handle demanding coding tasks? And will the savings survive a realistic working day, with people sharing the hardware?
We recently went through a real-world benchmarking exercise with AMD Instinct Coder and the latest GLM 5.3 model to answer exactly those questions. We measured responsiveness and throughput as concurrency increased, ran a sustained workload, and compared the cost versus hosted API prices.
The results make a strong case for local inference. One 8-GPU appliance comfortably supported 60 simulated developers and 15 coding agents during a five-hour test window, with a median first token of 406 milliseconds on warm developer requests.
At sustained utilization, API alternatives (including hosted versions of the same GLM 5.3 model) cost 3.3x to 20.6x as much.
The conclusion? Open models on local hardware can compete with the best… although spoiler alert: getting there took some tuning.
Building a benchmark for the real world #
Our tests used eight AMD Instinct MI325X GPUs, with GLM 5.3 in FP8 running through PaletteAI Inference Launchpad and vLLM 0.31.0.
Our goal was to simulate a real coding team in action, so we modeled Claude Code sessions for 75 simulated users, an average of 36 sessions open simultaneously. We modeled most users as human-style developers with high reasoning effort on, and around one in five as an agent harness with reasoning off, running short, tool-heavy sessions of 3–6 turns.
Half the developers are light users who start with 27K to 35K tokens of context, 40% are medium at 35K to 50K, 10% are heavy at 50K to 80K, and about one identity in five is an agent harness running short, tool-heavy sessions of 3 to 6 turns. Every turn grows the context with tool results (around 2,500 tokens at the median, twice as fast for heavy users and half as fast for light ones), and sessions compact at 167K the way Claude Code does, which keeps the average live prefix around 40K tokens.
We focused on warm turns: the second request in a conversation onward, when earlier context can be reused, because coding involves repeated exchanges over the same repository and conversation. In our run, about 91% of input tokens came from the prefix cache. Testing every turn as a fresh prompt would miss much of that behavior.
We also needed to decide what counted as “responsive”. None of the providers we checked published a first-token target we could adopt. Nielsen Norman Group’s usability guidance puts roughly ten seconds at the limit of sustained attention. We used that guidance alongside inference benchmarks to set our own bar: 95% of warm developer requests must start within five seconds, 99% within ten, and errors must stay below 1%. We wanted each user to be getting at least 30 tokens per second, even under contention from multiple concurrent sessions. And we checked that every hour after warmup to ensure this performance was sustainable for the full working day.
Those thresholds were our engineering judgment. They aren’t an industry standard or a claim that the coding task finishes within ten seconds. Following Google SRE’s guidance on latency percentiles, we wanted to account for the slower requests too.
The results: fast responses held up under load #
Our Instinct Coder + GLM 5.3 setup delivered extremely strong results on our benchmarks:
- 98% of warm developer requests started within five seconds.
- Answers streamed at 46 tokens per second per stream.
- We recorded zero request errors across 18,475 requests.
For fresh 10,000-token prompts, sent one at a time, we measured 0.59 seconds to the first token and 143.5 output tokens per second. That output rate exceeded the Claude and GPT endpoints. Raw throughput was higher with many short requests than with large token prompts: 3,205 output tokens per second across the appliance at 64 concurrent requests with 1,024-token inputs and outputs. We used random inputs to prevent cache reuse and reported the median of three repetitions.
Fast turns and high throughput are no good if the quality of the responses is poor. So it’s worth noting that GLM 5.3 scored 95.4% on Vals AI’s SWE-bench Verified evaluation, close to Claude Opus 5’s 97%, using the same harness.
We didn’t take this benchmark for granted — we ran our own quality checks during our tests too.
Before each run, a check gate asks the model a known-answer question, makes it call a tool and hides a needle in about 54K tokens of text. On top of that, we then run a 1,147-item accuracy set (HumanEval+, MBPP+, GSM8K and long-context retrieval) and a 69-task quality suite against the baseline, where the tuned setup must score above 90%, otherwise it gets rejected. Lastly, on the five-hour soak tests we also run the quality suite every 30 minutes under load, to make sure it continues to perform with no drift. This is important because for most of them we know the answers are right before the load starts, and we still have to prove that they stay right after hours with the GPUs and the model under heavy pressure.
The power of tuning: from a 15.9-second wait to 0.82 seconds #
The most important finding in our benchmark is just how much tuning matters.
Our first deployment wasn’t producing the great results we shared above. In the same 64-request burst test, tuning changed performance substantially:
That’s a 19x improvement in time to first token, and a 50% increase in throughput.
To achieve that, we made eight changes:
- Concurrent sequences: 128, was 32. +36% output and -27% median first token at 32 concurrent requests on short prompts, three fresh engine starts per side.
- Context window: 262K, was 1M. +36% GPU cache.
- GPU memory share: 0.85, was 0.80. +16% GPU cache.
- Multi-token prediction: 5 draft tokens. Turning it off cost 42% of output on 8K prompts at 32 concurrent requests, one run.
- fastokens tokenizer: on. Cached 50K-token follow-up: first token 112 ms to 75 ms at one stream.
- Batched tokens: 16,384. Two five-hour tuning runs in September against 4,096: +5% output.
- System memory tier, 128 GiB: on. With 80 users all working at once, more context than the GPU cache holds: +52% to +64% output.
- Sparse-attention refresh: every 8 layers, was 4. Ladder at 45 users, single runs: +34% output, p95 first token 5.1 s to 2.4 s.
These were separate experiments; their percentage gains don’t add together. But collectively, they deliver the huge improvement described above.
Note that we package the tuned configuration in the PaletteAI Inference Launchpad catalog so you can start from this work without having to do your own tuning!
You don’t need round-the-clock usage to save #
Instinct Coder is a hardware and software appliance. The back of a napkin numbers: $450,000 for hardware, $50,000 annually for software and $50,000 for operations, including power and cooling. Over three years, that nets out to about $20,800 a month, whether the appliance is busy or idle.
How does that stack up against hosted inference from the frontier?
We applied October 3rd 2026 API rates, including cache discounts, to the measured traffic during our test: 232 million input and 2.2 million output tokens per busy hour. The monthly figures extrapolate that rate.
Of course, comparing a flat cost against a per-hour charging model means it’s important to know how many hours a day you’ll be using the service. It changes the numbers massively!
We modeled several scenarios, but the two extremes are in the chart below: a dev team working one shift per working day, which works out to 168 hours per month; and a ‘round the clock’ scenario where users are always connected, 730 hours per month (this would suit long-running agent workloads or multiple teams on different time zones).
Even at eight hours across 21 working days, local inference with GLM 5.3 on the Instinct Coder box costs less than three of the five alternatives — a third less than Opus, and less than half the price of GPT-5.5. Only Z.ai with GLM and Sonnet remain cheaper at that utilization.
Above roughly 222 busy hours a month, the Instinct Coder appliance costs less than any of the hosted models we checked, and if you were heavy users of Opus or GPT-5.5, you would be saving 80% or more.
Try for yourself #
We undertook this benchmarking to prove with some degree of rigor that local inference can pass muster not just for cost savings, but throughput, responsiveness and coding quality. It did: we’ve shown that local inference can support a busy development team with fast responses and substantial savings, even if you don’t use it round the clock. Along the way we proved just how important it is to tune how you serve each model, though.
Your next step? We offer a TCO calculator and a sandbox where you can try a real Instinct Coder for yourself, to replicate our tests with your own workloads. Learn more here.