You Can't Version-Pin Somebody Else's Model

A data company moved its AI enrichment pipeline for 10,000 business profiles off hosted frontier models and onto owned hardware — three machines with four 32GB AMD workstation cards each for $32,201 — after a vendor silently changed model behavior, causing refusals on legal firearms-related businesses that returned HTTP 200 while enrichment quietly stopped. The local setup processes over 175 validated jobs a minute versus 12 for an earlier six-Mac-Studio attempt, using a small dense open-weights model for volume and big-memory Macs for auditing and complex extractions. Separately, a from-scratch C inference engine streams mixture-of-experts weights from NVMe to run a 48GB model on a 32GB laptop at 24 tokens/s, versus 2.8 for llama.cpp.

The argument for local inference used to be privacy, and it used to be unconvincing. This week it turned into something harder to wave away: throughput, cost, and a dependency you cannot pin. 1. The Vendor Didn't Raise the Price. It Changed the Answers. A data company running an AI enrichment pipeline over 10,000 business profiles just moved all of its inference onto hardware it owns https://nicholasreese.com/why-i-started-moving-inference-in-house/ — three machines, four 32GB AMD workstation cards each, $32,201 all in. What pushed them was not the invoice. A frontier vendor had cut prices sharply, they went all in, and the numbers were fine. Then the model began refusing work it had done the week before: declining to identify a licensed firearms dealer, declining to read an ammunition retailer's website, asking for more context before summarizing a shooting range's posted policies. Read that as an engineer and it is a dependency problem wearing a policy costume. Their site is a directory of gun stores, instructors and gunsmiths — legal businesses in a policy-adjacent domain — and the capability they had built on narrowed under them silently. No version to pin, no changelog, no rollback. A refusal is an outage with no stack trace: the call returns 200, the pipeline keeps running, and the data quietly stops being enriched. The economics were the easy part. Their first local attempt, six Mac Studios, managed 12 validated jobs a minute; scaling that way meant another 41 to 56 machines and roughly $280,000 to $380,000. The twelve workstation cards do more than 175 a minute for a tenth of that, and absorb the CI/CD pipeline besides. The architecture that emerged is the interesting part: a small dense open-weights model grinds volume while the big-memory Macs audit its output and handle complex extractions. Cheap determination at the bottom, expensive judgment on top. Why it matters: - For ICs: Add refusal-rate and schema-conformance checks to your eval suite and run them on a schedule, not only at deploy. A behavior regression you cannot see is one you will ship. - For leaders: "Which model version are we on?" should have a precise answer. If the honest answer is "whatever the endpoint served today," you have an unmanaged dependency in production. - For founders: If your product sits near a policy boundary — firearms, medical, legal, adult, political — hosted inference is strategic risk, not a cost line. Price the exit before you need it. - The general pattern: any layer you don't control can change semantics without changing its interface. We learned that with SaaS APIs; models are the same lesson at higher volatility. 2. An 80-Billion-Parameter Model Now Runs on a 32GB Laptop The usual objection to owning your inference is memory, and that ceiling is moving. A from-scratch inference engine written in C with no dependencies beyond libc https://github.com/prabanta-dev/janas runs a 48GB mixture-of-experts model on a 32GB laptop by leaving the weights on disk, streaming the experts each token needs off an NVMe drive, and keeping the ones that keep recurring in memory. It writes 24 tokens a second where llama.cpp manages 2.8, and reads prompts at 62 against 16 — not a marginal win, a different regime. The mechanism generalizes. In a sparse mixture-of-experts model only a few experts fire per token, so what must fit in RAM is the working set, not the parameter count. That converts a memory-capacity problem, where consumer hardware hits a wall, into a disk-bandwidth problem, where a cheap SSD reads at 5.7 GB/s. The same reframing drives research on aggregating idle Apple Silicon GPUs across a fleet of Macs into a virtual inference cloud https://machinelearning.apple.com/research/distributed-gpu-cloud : that capacity is already bought, already depreciating, and mostly asleep. The honest caveat: the engine calls itself an experiment, not a product. It fills free memory with its expert cache, reads the disk flat out with direct I/O, and has once taken a desktop down with a driver reset. That is roughly where llama.cpp sat in 2023, which is the point — capability arrives rough, then gets boring. Why it matters: - For ICs: Re-measure. The "this needs an 80GB card" assumption in your head is probably a year stale, and parameter count is no longer a proxy for hardware requirements. - For leaders: Before approving GPU spend, audit the fleet you own. Idle developer machines and build runners are capacity already paid for. - For founders: A local-first tier is becoming a real product option rather than a demo, which changes what you can promise about data residency. 3. The Fastest Model for the Job Has One Million Parameters The reflex when a task involves text is to call a frontier model. A new OCR package built around a one-million-parameter recognition model https://www.datalab.to/blog/indra-ocr reads 49 pages a second on 32 CPU cores — 230 milliseconds a page, 11 to 17 times faster than Tesseract and PaddleOCR's text models while beating both on accuracy, and roughly 180 times faster per page than a full document pipeline it also outscores on layout and tables. No GPU. The rationale is refreshingly unfashionable: for classification and segmentation, speed matters more than perfection, and good-enough text at high throughput unlocks work a slow, accurate engine never will. A second demonstration makes the point from a stranger angle. Take a multimodal embedding model never trained for speech recognition, train a small reader to map audio slices into its text space, match each slice against the nearest of 229,000 English words, and you get an hour of audio transcribed in about 13 seconds on a base MacBook Air https://eg2.haku.sh/ , entirely on device. Word error rate is about 5% against 2–3% for leading speech models. Training the reader took under two hours on eight GPUs. Both results say the same thing: we have been buying accuracy we do not need, at a latency and cost that do not survive volume. Pick the error budget first, then the smallest model that clears it. That decision, not the model roster, is where the engineering is. Why it matters: - For ICs: Benchmark a small specialist against your frontier call on your own data before assuming you need the big one. Many pipelines are paying frontier prices for a bounding box. - For leaders: Ask what accuracy each AI feature actually requires. An unspecified error budget defaults to the most expensive model available. - For founders: CPU-only inference means no GPU queue and no per-token floor on your margins. That is a pricing advantage competitors routing to an API cannot match. The Verdict: Real or Hype? In-house inference for high-volume pipelines → Real. When unit economics and dependency control point the same way, the build-versus-rent math has flipped; the open question is whether you want the operational burden. Larger-than-memory models on consumer hardware → Real but early. The technique is sound and the numbers are not close, but the implementations will eat your desktop before they stop being experiments. Task-sized small models replacing frontier calls → Real but unevenly distributed. Teams that measure their error budget already ship this; teams routing everything through one endpoint haven't noticed the option exists.