Practical Advice on Local Inference and Cloud LLMs - August 2026

A developer's two-week experiment with an M5 Max MacBook Pro with 128 GB of unified memory found that DeepSeek V4 Flash weights ran at 6 tokens per second under mainline llama.cpp but 30 to 40 tokens per second under the ds4 engine, a fivefold speedup from software alone. The developer routes all cloud workloads through Vercel's AI Gateway, which undercuts DeepSeek's first-party API pricing at $0.09 per million input tokens and $0.18 out versus $0.14 and $0.28, and offers more consistent latency. The laptop serves as the only zero-data-egress environment for sensitive code, while the gateway fronts eight providers for other tasks.

Earlier this year I bought an M5 Max MacBook Pro with 128 GB of unified memory. It was a deliberate investment in learning local inference: I wanted to run serious open-weight models on my own machine and find out, with my own measurements, where local ends and cloud begins. Two weeks in, I have answers I trust. The full measurements live in The Price Floor /blog/the-price-floor , a much more detailed drawing of the whole experiment. This post is the practical summary: what I actually run where, and why, as of August 2026. The date is in the title because half of these facts will move. What the 128 GB MacBook is actually for The headline result from the experiment: the same DeepSeek V4 Flash weights ran at 6 tokens per second under mainline llama.cpp and at 30 to 40 under ds4 https://github.com/antirez/ds4 , a small engine written specifically for that model family. Same laptop, same GPU, five times the speed from software alone. If you take one thing from this post, take that. Before you conclude your hardware is too slow, or buy different hardware, try a different engine. At 30 to 40 tokens per second, the laptop is genuinely useful. A person reads at roughly five or six tokens per second, so the model writes several times faster than I can read it. Claude Code points at the local server directly, because ds4 serves the Anthropic-style API that Claude Code expects, and the whole setup works on an airplane. What the laptop earns with that speed is a specific job: it is the only place in my setup with zero data egress. Client code, proprietary work, anything I'm not free to send to a third party, runs there and nowhere else. That matters more than it might seem, because the cheapest capable cloud option stores data on servers in China with no opt-out from model-improvement use in its terms. Cheap and private are different products. The laptop is a personal instrument. It serves one session at a time with no batching, and a 97 GB model leaves the machine with limited headroom for anything else heavy. Treated as a personal instrument, it's excellent. The education alone justified the purchase: after two weeks of running models directly, I understand engines, quantization, caching, and serving in a way that reading never produced. Why Vercel AI Gateway won my DeepSeek workloads, for now For everything that doesn't need to stay on the machine, I expected to use DeepSeek's own API. Instead, everything routes through Vercel's AI Gateway https://vercel.com/ai-gateway , a router that fronts eight providers hosting the same open model. Five measured reasons: - It's cheaper than DeepSeek first-party for the same model. $0.09 per million input tokens and $0.18 out, against DeepSeek's $0.14 and $0.28, with the identical $0.0028 cached-input price. A router undercutting the model's own maker is a strange fact about open weights, and it's true today. - Claude Code connects with two environment variables. The Gateway speaks Anthropic's /v1/messages protocol natively, so there's no proxy or translation layer: export ANTHROPIC BASE URL="https://ai-gateway.vercel.sh" export ANTHROPIC API KEY="$AI GATEWAY API KEY" claude Tool calling works. Prompt caching works and is reported per request. - Its latency was the most consistent I measured. Across my streamed trials the Gateway's first token arrived in 0.35 to 0.63 seconds every time, while a direct provider serving the same checkpoint threw 8 and 9 second outliers. Median throughput was identical, about 73 tokens per second either way. Consistency is what you feel in interactive work. - Eight providers behind one bill mattered the week I shopped. I tried to buy fast inference from three vendors in one evening and every one turned me away: Makora's $20 and $200 plans both showed sold out, Cerebras Code Pro and Code Max were sold out with restocks going the same way, and Groq's paid Developer tier had been closed to upgrades for months. Single-provider dependence is a real availability risk right now, and the Gateway routes around a provider having a bad day without me noticing. - Zero-data-retention options exist per provider for work that can leave my machine but shouldn't be retained. The "for now" is load-bearing. Providers reprice monthly, the Gateway's routing changes under the hood, and the 30 to 45 percent gap I measured between listed and actual throughput says published numbers age fast. I plan to re-run the same benchmarks quarterly, and the scripts made that cheap. Where each workload goes | Work | Where it runs | Why | |---|---|---| | Client and proprietary code | MacBook, local ds4 | Nothing leaves the machine | | Personal projects, open source | V4 Flash via AI Gateway | Cheapest capable option, caching included | | Fast iteration bursts | Fast metered silicon, when a seat exists | Wrong answers are cheap, speed compounds | | Hardest problems | Frontier API | Quality per attempt beats price per token | The fast-silicon lane deserves a note, because it barely exists as a retail product right now. Groq and Cerebras will serve you 300 to 2,100 tokens per second, but on smaller models, and as of this writing the flat-rate plans are closed everywhere I looked: Makora, Cerebras Code, and Groq's Developer tier alike. The metered tiers at Groq and Cerebras remain open, and metered is what I'd reach for when a burst of boilerplate or refactoring justifies it. The closest thing I've found to fast capacity you can actually buy today is Baseten https://www.baseten.co/pricing/ . Their per-token model API serves V4 Flash 0731 at $0.13 in and $0.26 out, a hair under DeepSeek's own prices, though the cached-input rate is $0.028 per million, ten times the Gateway's, which keeps the Gateway ahead for agent workloads shaped like mine. The more interesting product is dedicated deployments: you rent the GPU itself by the minute, an H100 for roughly $6.50 an hour or a B200 for roughly $10, and serve any open model on it with no seats to sell out. It's also where brand-new open weights tend to land first. Thinking Machines released Inkling https://thinkingmachines.ai/news/introducing-inkling/ , their 975-billion-parameter Apache 2.0 model, in mid-July, and it was on Baseten the same day. One account covers both experiments, and that combination is next on my list. The one number to check before comparing providers Coding agents re-send their whole context on every turn: system prompt, tool definitions, conversation history. Providers cache the repeated prefix and bill it at a discounted cache-hit rate. In my own two-week ledger, 96.6 percent of all input tokens were cache reads. At that ratio, the cache-hit price effectively is the price, and vendors set it very differently: DeepSeek and the Gateway charge $0.0028 per million cached tokens, one direct reseller charges ten times that for identical throughput, and some fast providers only discount hits by half. So before comparing providers, pull your own usage records and compute your own cache-read share. Mine made two weeks of heavy daily agent use cost about $17 a month through the Gateway. The same tokens through the priciest rate card I modeled would have cost over $2,000. The spread between those numbers is almost entirely the cache-hit line, and most pricing pages don't print it. If you're tempted by hardware Four things I verified the hard way, condensed: The engine tax is real at every scale. The same four-GPU workstation benchmarks anywhere from 25 to 700 tokens per second depending on serving software. Software maturity is the biggest risk in any hardware purchase right now. Measure before believing. Published throughput ran 30 to 45 percent above what my own clock recorded, and run-to-run variance exceeded the differences between providers. The break-even is knowable. A four-card Blackwell build lands near $34k, which amortizes to roughly $1,160 a month over three years. Below that in monthly model spend, the API wins on cost. Above it, and the sold-out market adds its own argument, ownership starts making sense. Write the parts list the month you buy. GPU list prices moved 55 percent in sixteen months and the serving stacks are improving weekly. My own build plan has a date on it, and no components. What I'd re-check next quarter Three things move fastest: whether the open-source engines close their gap to the hardware's theoretical ceiling on new model architectures, whether GDDR7 supply lets GPU prices retrace, and whether any vendor reopens a flat-rate fast tier. The method outlasts all of it. Pull your own ledger, weight prices by your own token mix, and time the providers with your own clock. That habit cost me one Saturday and now saves me money every day.