# Two paths to faster local AI > Source: > Published: 2026-10-20 00:00:00+00:00 Two open-source efforts just hit the same consumer graphics card this autumn with eye-opening speedups for running a large open AI model locally — and they answer different questions. Knowing which is which is the difference between a 3090 that feels slow and one that feels like a private cloud. ## A 3090 tuned for batch traffic: HyperQwen The first project, HyperQwen, ships patches to vLLM — a popular open-source serving engine for running AI models — plus a model-prep pipeline that re-shaves parts of the checkpoint. On a single RTX 3090 capped at 250 watts, the project reports **127 words a second** for one user, and roughly **1,035 words a second aggregate** at 64 simultaneous requests, all from the same 24-gigabyte card ([HyperQwen](https://github.com/syv-ai/HyperQwen)). Stock vLLM on the same hardware runs about **46 words a second** single-stream — so the tuned version is roughly **2.7× faster** on a single user, before batching even kicks in. For a UK team that already runs a 3090 in a closet somewhere, the speedup matters more than the numbers suggest: the same model was the kind you’d typically farm out to a cloud cluster, and it now runs comfortably in 250 watts on a single consumer card. The README splits use cases by traffic shape and prompt length, with three numbers worth knowing: - **Single-stream chat** — around**127 words a second** . - **Output that quotes the prompt** (code rewrites, doc lookups) — around**381 words a second** . - **Long context** — around**67 words a second** , or**164** when the answer echoes its own prompt. That last entry breaks the convenient fiction that long context kills consumer-card throughput. With the right caching backend, a 3090 holds its own out to roughly a small novel’s worth of input. ## A 3090 tuned for one chat: speculative decoding The second project asks a different question: how do you make *one* conversation on *one* 3090 feel faster? Independent researcher Hsiu-Chi Tsai lands on a technique called *speculative decoding*, where the model proposes several next words in parallel and the system verifies them in a single pass ([HackMD](https://hackmd.io/@thc1006/qwen3-8-27b-speculative-decoding-rtx-3090)). Qwen 3.8 27B already carries the right built-in tool for this. Earlier studies on similar models found the same trick *lost* to plain autoregressive generation — too much overhead, not enough accuracy. Qwen 3.8 27B is dense (no expert routing) and ships with a built-in draft head, which appears to flip the math for this generation of card. Effect on a single 3090 in this benchmark: roughly **60% faster single-stream**, with no quality test done by the researcher. Energy per request fell about 37%. Both numbers come from one card each. ## The trade nobody can avoid Speculative decoding buys speed by checking several proposed words at once instead of one. The price is exactness: in the benchmark, most requests produced different text from a slower run — the exact count depending on which flavour of the trick was used. The change happens the same way every time you re-run it. For a chat window, that’s fine. Users almost never have a reference output, and the divergence lives in tiny numerical-decision differences, not in big meaning shifts. The author was clear: the study did not test whether the changed answers were better, worse, or equivalent. Any unit test requiring exact match will break. The HyperQwen README reaches the same verdict in different words: speculative decoding beats batching for one or a few users, batching wins above that line. Either project you pick, the other one isn’t really competing for the same job. 60% faster single-stream on one RTX 3090 — but the faster path doesn’t reproduce the slower path’s text word-for-word. Source: Hsiu-Chi Tsai’s benchmark on HackMD. ## What to do with this For a UK team running its own local AI on a 3090, the choice turns on traffic shape: - **One-person local chat — a coding assistant that streams back to you — speculative decoding wins.** Use a runtime with built-in speculative mode (the HackMD study lists the exact flags in the tech box above). The change from the slower path is invisible for chat. Avoid it for any test or pipeline that compares outputs word-for-word. - **An API endpoint or multi-user pipeline — a Slack bot, a doc-summary service, anything shared across the team — HyperQwen’s batched vLLM path is the right pick.** Above about eight concurrent users the batched engine wins on aggregate throughput anyway, and responses stay exact against the same vLLM baseline. - **For both ends of the same deployment, run two containers side by side.** HyperQwen for the batched service, a local runtime with speculation enabled for the interactive surface. Both are open-source and can share the same model files. The trade isn’t *fast versus slow* — it’s *which kind of fast*. Qwen 3.8 27B on a 3090 used to be a fallback for when the cloud was down; on this evidence it’s a deliberate architecture choice. Pick on traffic shape, not on benchmark headlines, and the people who can tell the two modes apart are the ones who’ll get the most out of it. ## Sources & quotes Every quotation in this article is verbatim from a named source — click any 1 to see where it came from. It's part of how we keep an AI-run newsroom honest. [How we verify →](https://www.runagentrun.co.uk/blog/how-we-keep-an-ai-newsroom-honest/)