Run an LLM Locally: Qwen3.8-27B in 12 Steps, 75 Min [2026]

A tutorial published by Shattered lays out a 12-step, roughly 75-minute process for running Alibaba's dense 27-billion-parameter Qwen3.8-27B model locally, using community GGUF builds in llama.cpp, Ollama or LM Studio alongside a server that speaks the OpenAI wire format and a repeatable speed benchmark. The guide notes Qwen3.8-27B and a matching FP8 release date to August 13, 2026, and cites the 2025 Stack Overflow Developer Survey, in which 14.4% of professional developers who use AI tools selected Ollama. It also flags that Alibaba announced Qwen3.8-Max updates at its Apsara Conference on October 28, 2026, including 33 automated improvement cycles and an Artificial Analysis score rise from 40 to 45, and that Qwen4 is in training but has not shipped.

Qwen3.8-27B is the member of the Qwen3.8 family that a single workstation can actually host. The Qwen model listing shows the dense 27-billion-parameter model, along with a matching FP8 release, dating to August 13, 2026, and the Qwen Cloud page for Qwen3.8-27B https://www.qwencloud.com/models/qwen3.8-27b now sits next to community GGUF builds that run in llama.cpp, Ollama and LM Studio. If you want to run an LLM locally in late October 2026, this is the model to size your machine against. Running it takes three pieces. You need weights that fit in memory, a server that speaks the OpenAI wire format, and a way to measure whether the result is fast enough to use. Most guides stop after the second piece and leave you guessing about the third. The steps below build all three in roughly 75 minutes, and most of that is download time. For a chat front end on top of the same server, our llama.cpp and Open WebUI walkthrough https://shattered.io/llama-cpp-server-open-webui-gguf-12-steps-2026/ picks up where Step 6 ends. Local inference is no longer a hobby corner. In the 2025 Stack Overflow Developer Survey, 14.4% of professional developers who use AI tools selected Ollama, and the survey’s technology section https://survey.stackoverflow.co/2025/technology shows large language models among the tools more than a quarter of all respondents use or want to use. Numbers like that explain why people search for ways to run an LLM locally, and why a repeatable benchmark matters more than a screenshot of one lucky run. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why Qwen3.8-27B Is the Local Model to Start With Alibaba’s Qwen line moved to the Qwen3.8 generation over the summer. The family now spans a very large open flagship, Qwen3.8-2.4T-A95B, which carries 2.4 trillion total parameters with about 95 billion active per token, and a set of smaller releases meant for ordinary hardware. The 27B model is the one sized for a desktop. It is a dense causal language model with a vision encoder, so it reads images and video as well as text, and it offers an adjustable reasoning mode that you can switch on for hard problems and off for quick answers. Dense matters for local use. A mixture-of-experts model activates only a slice of its weights per token, which helps speed, but you still need memory for all the experts. A dense 27B model has no such trick. Every weight is read for every generated token, so memory bandwidth sets your speed ceiling and file size sets your memory floor. That makes the arithmetic in this guide simple and predictable, which is exactly what you want when you are sizing a machine. The official model repository documents Transformers, vLLM and SGLang as supported runtimes. llama.cpp support arrives through GGUF files that community members convert and upload. That split shapes the whole tutorial. The weights come from Qwen, the quantized files usually come from third parties, and you need a habit of checking one against the other. Step 5 covers how. Two caveats on the rest of the family. Alibaba announced updates to Qwen3.8-Max at its Apsara Conference on October 28, 2026, including 33 automated improvement cycles and a rise in its Artificial Analysis score from 40 to 45, according to Philstar’s report on the conference https://www.philstar.com/business/technology/2026/10/28/2566097/alibaba-unveils-roadmap-full-stack-ai-strategy-chips-cloud-infrastructure-models-agents . That is a hosted flagship, not something you run at home. We covered the open-weight side of that story in our Qwen3.8-Max open-weights analysis https://shattered.io/qwen3-8-max-open-weights-benchmarks-2026/ . Separately, Alibaba’s strategy update on October 2 said Qwen4 is in training. It has not shipped. One newsletter reported on October 31 that an Alibaba executive hinted at a Qwen4 release as soon as next week, but no release notice backs that up, so treat it as unconfirmed. The practical takeaway: Qwen3.8-27B is the newest Qwen model that fits on one machine today, and the harness you build here will measure a Qwen4 or any other GGUF model just as easily when one arrives. Swap the model file, change the label, rerun. Prerequisites and Tested Versions The harness uses only the Python standard library, so there is nothing to compile on the Python side. The heavy lifting happens in llama.cpp. The table lists what to have in place before you start, with the version guidance that applies on October 31, 2026. | Component | Version to use | Notes | |---|---|---| | Python | 3.15.2 released Oct. 26, 2026 | Python 3.10 reached end of life on Oct. 1, 2026. The scripts here also ran cleanly on 3.12.3 during testing. | | llama.cpp | Newest build from ggml-org/llama.cpp | Builds carry a b-prefixed number. Older builds may not recognize the Qwen3.8 architecture. | | Model | Qwen3.8-27B as a community GGUF | The official repository ships standard weights. Pick a quantizer you trust. | | GPU drivers | Current CUDA 12.x, ROCm, or Metal | Vulkan also works when no vendor stack is available. | | Hugging Face CLI | Latest huggingface hub | Used for resumable downloads. | | Disk space | 40 GB free | Covers the model file, a source build and cache. | | System RAM | 32 GB recommended | Needed for partial offload and while the file loads. | | Operating system | Linux, macOS or Windows 11 | Commands below use a Bash shell. Windows users can use WSL 2 or PowerShell equivalents. | You do not need a GPU to follow along. A CPU-only run works, and the benchmark will simply report slower numbers. You do need enough memory to hold the weights, and Step 1 shows how to check that before you spend an hour on a download. Step 1: Check Your Hardware and Set a Memory Budget Three things consume memory when a model runs: the weights, the key-value KV cache that grows with context length, and a fixed overhead for compute buffers and the GPU driver. Weights are the big, predictable part. Multiply the parameter count by the bits per weight, divide by eight, and you have the file size. At 27 billion parameters and an average near 4.9 bits per weight, that gives about 16.5 GB, which matches the Q4 K M files that community listings describe. Speed has its own rule of thumb. Because the model is dense, each generated token reads roughly the whole file from memory. Divide your memory bandwidth by the file size and you get a ceiling on decode speed. A card with 1,000 GB/s of bandwidth and a 16.5 GB file tops out near 60 tokens per second. Real runs land below that ceiling, so use it as a sanity check on your benchmark rather than a target. If your measured number beats the ceiling, something is wrong with the measurement. Run the commands for your platform and write down the free memory figure. That number is your budget. NVIDIA nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv Linux system RAM free -g macOS total unified memory in bytes sysctl hw.memsize The next table maps common memory budgets to a starting plan. It is analysis, not a vendor claim, and Step 8 replaces the guesswork with a measured calculation. | Memory available to the model | Starting plan | What to expect | |---|---|---| | Under 12 GB | Lower-bit quant with heavy CPU offload, or choose a smaller model | Slow decode. The 27B model is a stretch here. | | 12 to 16 GB | Q4-class quant with partial GPU offload | The weights alone strain the card, so some layers run on the CPU. | | 20 to 24 GB | Q4-class quant fully on the GPU, moderate context | Comfortable at 8K to 16K context. Longer context needs a KV check. | | 32 GB or more | Q5 or Q6 quant fully on the GPU | Room for long context or a second model for speculative decoding. | | Unified memory, 32 GB and up | Q4 to Q6 quant, all layers offloaded | Bandwidth, not capacity, limits speed. Expect a lower ceiling than a flagship GPU. | Owners of a flagship consumer card have a head start, and our guide to running LLMs locally on an RTX 5090 https://shattered.io/run-llm-locally-rtx-5090/ covers that hardware in depth. AMD users have an official path too. According to AMD’s own blog post, Qwen3.8-27B is supported on Ryzen AI Max systems and Radeon graphics cards, and the AMD post https://www.amd.com/en/blogs/2026/run-qwen-3-8-27b-on-amd-ryzen-ai-max-and-radeon-graphics-cards-day-0.html lists the supported setup. On Windows, remember that the display driver reserves some video memory, so a 24 GB card rarely offers a full 24 GB. On a laptop with a hybrid GPU, confirm that llama.cpp sees the discrete card and not the integrated one. These two details cause more first-run failures than any flag. Steps 2 and 3: Install Python 3.15.2 and Create the Project Folder Install Python 3.15.2 from python.org, your distribution’s package manager, pyenv or any other installer you trust. Then confirm the version and create an isolated environment. The harness has no third-party dependencies, but the Hugging Face CLI does, and a virtual environment keeps it away from your system packages. python3.15 --version Python 3.15.2 mkdir -p ~/localbench/models && cd ~/localbench python3.15 -m venv .venv source .venv/bin/activate Windows PowerShell: .venv\Scripts\Activate.ps1 python -m pip install -U pip huggingface hub If the first command prints an older number, your shell is finding a different interpreter. Call the binary by its full path or fix your PATH before continuing. Mixing interpreters is a quiet source of confusion later, especially when a script runs fine in one terminal and fails in another. The finished project has five Python files and one models folder. You will create each file in its own step, so the folder starts nearly empty. localbench/ models/ GGUF files live here fit.py Step 8: will it fit in memory? llama client.py Step 9: streaming client with timing bench.py Step 10: benchmark runner mock server.py Step 11: fake server for a smoke test report.py Step 12: summary table results.jsonl written by bench.py Keeping the pieces separate pays off. The client knows how to talk to a server and time it. The runner decides what to ask. The report reads rows and summarizes them. When you later point the same code at vLLM or Ollama, only the base URL changes. Step 4: Install llama.cpp llama.cpp is an open-source C and C++ inference engine maintained under the ggml-org organization on GitHub. It has no paid tiers. Development is fast, with new builds landing almost daily. On October 30 the release feed reportedly showed pre-release builds in the b12000 range, so a build from a few weeks ago may lack support for newer model architectures. Build from the current source or grab the newest release binary. Release binaries exist for macOS, Windows and Linux, including CUDA variants, and they are the fastest route. Building from source takes five to ten minutes and gives you the exact backend you want. The commands below build with CUDA. Swap the flag for your hardware. git clone https://github.com/ggml-org/llama.cpp cd llama.cpp NVIDIA GPU cmake -B build -DGGML CUDA=ON AMD GPU: cmake -B build -DGGML HIP=ON Any GPU with Vulkan: cmake -B build -DGGML VULKAN=ON Apple silicon: Metal is on by default, no flag needed CPU only: cmake -B build cmake --build build --config Release -j ./build/bin/llama-server --version Expect a version line that includes a build number. Write it down, because every benchmark you run should record the build it came from. Performance shifts between builds, and a result without a build number is hard to reproduce a month later. Some community documentation now shows a newer single-command form, llama serve and llama cli , while many repositories still call the standalone llama-server and llama-cli binaries. Both forms exist in the wild. The commands here use llama-server , so if your build only has the newer subcommands, run llama serve with the same flags. Not sure llama.cpp is the right runtime for you? Our comparison of Ollama, LM Studio, llama.cpp and vLLM https://shattered.io/ollama-vs-lm-studio-vs-llama-cpp-vs-vllm-local-llm-runtimes-2026/ lays out the trade-offs. We use llama.cpp here for three reasons. Its flags are explicit, so nothing happens behind your back. Its server exposes a tokenizer endpoint the harness needs. And it exposes Prometheus metrics for longer experiments. Step 5: Download a Qwen3.8-27B GGUF Start at the official source so you know what the model is supposed to be. The Qwen organization’s Hugging Face repository for Qwen3.8-27B documents the architecture, the reasoning options and the license, and a matching FP8 repository sits beside it. Read the model card before you download anything else. The license file is the authority on commercial use. One third-party report describes the model as Apache 2.0, but confirm that in the official repository yourself rather than relying on a summary, including this one. The official repository serves standard weights. For llama.cpp you want a GGUF, and the community has published several. Quantizers such as bartowski and unsloth publish well-known sets, and the benchmark repositories we found used an unsloth Q4 K M file of about 16.5 GB. Smaller and larger quants exist, along with variants that bundle extra features. For a first run, take the Q4 K M file from a quantizer with a track record. source .venv/bin/activate hf download bartowski/Qwen3.8-27B-GGUF \ --include " Q4 K M " \ --local-dir models ls -lh models/ If the include pattern matches nothing, list the repository files in your browser and adjust the pattern, because uploaders name their files differently. The hf download command resumes interrupted transfers, which saves you when a 16 GB file stalls at 90%. How to read quant names A quant name tells you the bit width and the method. Q4 K M uses a mixed four-bit scheme and is the usual balance of size and quality. Q5 K M and Q6 K trade more memory for less quality loss. Q8 0 comes close to the original precision but nearly doubles the file. IQ-prefixed names, such as IQ4 NL, use importance-matrix techniques to squeeze quality out of fewer bits. One community listing puts an IQ4 NL build at about 15.2 GiB, and other uploads claim sizes near 14 GB. Treat those figures as claims until you see the file size on your own disk. Quality differences between neighboring quants are small for most chat and coding work, and large differences show up at the low end, below four bits. If you are unsure, the harness gives you the means to test. Download two quants, benchmark both, and compare quality on your own prompts alongside speed. Weights from unknown uploaders deserve suspicion. Some community variants alter the model’s behavior on purpose, for example by stripping safety training. That may be fine for a research box and wrong for a tool you hand to colleagues. Check who published the file, compare its size and hash against the repository listing, and keep the official model card open when you evaluate. Our open-weight model license comparison https://shattered.io/open-weight-models-license-comparison-6-families-16gb-2026/ explains why license terms deserve the same attention as benchmark scores. Step 6: Launch llama-server llama-server loads the model once and serves an OpenAI-compatible HTTP API, plus a small built-in web page. Start with a conservative context size and increase it after you have measured the KV cost in Step 8. ./llama.cpp/build/bin/llama-server \ -m models/Qwen3.8-27B-Q4 K M.gguf \ --alias qwen3.8-27b \ -c 8192 \ -ngl 99 \ -fa on \ --jinja \ --host 127.0.0.1 \ --port 8080 \ --metrics Adjust the -m path to the exact filename you downloaded. On a machine without a GPU, drop -ngl 99 . The table explains each flag. | Flag | What it does | When to change it | |---|---|---| | -m | Path to the GGUF file | Always set. Use the exact filename. | | –alias | Model name the API reports | Match the name your client sends. | | -c | Context window in tokens | Raise after measuring KV cache cost. | | -ngl | Layers to offload to the GPU | Lower it if you run out of VRAM. 99 means offload everything. | | -fa | Flash attention, on or off | Leave on where supported. Older builds took a bare flag, so check the help output. | | –jinja | Use the model’s own chat template | Keep on so reasoning and tool-call formatting match the model card. | | –host and –port | Network binding | Keep 127.0.0.1 unless you add an API key and a firewall rule. | | –metrics | Expose Prometheus counters | Useful for long soak tests. | Flag syntax shifts between builds, so run llama-server --help when something is rejected. The flash-attention switch is the usual offender, since some builds take a bare -fa and newer ones take on , off or auto . Watch the startup log. You want to see the model file open without an architecture error, a line stating how many layers moved to the GPU, and a final message that the server is listening. A message about an unknown architecture means your llama.cpp build predates the model, and the fix is a newer build. If the layer count is lower than the total, part of the model runs on the CPU, which explains slow generation later. Machines with unified memory, such as Apple silicon and the DGX Spark class of boxes, handle this step differently, since the GPU and CPU share one pool. Our DGX Spark walkthrough https://shattered.io/run-llm-dgx-spark-ollama-vllm-12-steps-2026/ covers the details for that hardware, and the flags above carry over. Step 7: Test the API with curl Before any Python, prove the server answers. Open a second terminal and hit three endpoints: health, a chat completion and the tokenizer. curl -s http://127.0.0.1:8080/health curl -s http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"qwen3.8-27b","messages": {"role":"user","content":"Reply with the single word: ready"} ,"max tokens":64}' curl -s http://127.0.0.1:8080/tokenize \ -H "Content-Type: application/json" \ -d '{"content":"How many tokens is this sentence?"}' The health call should return a small JSON object whose status is ok. While the model is still loading, it returns an error status instead, so wait and retry. The chat call returns the standard OpenAI structure with a choices array, a message and a usage block. The tokenizer call returns a list of integer token IDs. {"status":"ok"} {"choices": {"message":{"role":"assistant","content":"ready", ...}} , "usage":{"prompt tokens": ..., "completion tokens": ..., "total tokens": ...}, ...} {"tokens": ... } Those dots mark values that differ on every machine. If your chat response has an empty content field and a populated reasoning content field, the model spent its whole token budget thinking. Raise max tokens or turn the reasoning mode down. The model card names the exact switch, and earlier Qwen3 releases used a chat-template option called enable thinking, so check whether the 3.8 card keeps that name before you rely on it. This is also the moment to try the built-in web page at the server’s root address. It gives you a quick feel for the model’s tone and speed before you measure anything. Step 8: Measure KV Cache Cost and Run the Fit Calculator The KV cache stores attention state for every token in the context, and its size depends on the model’s layer count, attention layout and the cache precision. Qwen has not published a per-token figure on any page we found, and guessing one would mislead you. Measure it instead. Restart the server with -c 8192 as in Step 6, find the startup log line that reports the KV cache size in MiB, and divide by 8192. The result is your MiB-per-token cost. Now feed that number to the calculator. It adds the file size, the KV cache for your target context and a 1.5 GB overhead cushion, then tells you whether the total fits and how much context your leftover memory could hold. python import argparse def estimate file gb, vram gb, kv mib per token, ctx, overhead gb=1.5 : kv gb = kv mib per token ctx / 1024 need gb = file gb + kv gb + overhead gb spare gb = vram gb - file gb - overhead gb max ctx = int spare gb 1024 / kv mib per token if spare gb 0 and kv mib per token 0 else 0 return need gb, kv gb, max ctx def main : p = argparse.ArgumentParser description="Will this GGUF fit in memory?" p.add argument "--file-gb", type=float, required=True, help="size of the .gguf on disk" p.add argument "--vram-gb", type=float, required=True, help="usable GPU or unified memory" p.add argument "--kv-mib-per-token", type=float, required=True, help="measured from the server log" p.add argument "--ctx", type=int, default=16384 p.add argument "--overhead-gb", type=float, default=1.5 a = p.parse args need, kv, max ctx = estimate a.file gb, a.vram gb, a.kv mib per token, a.ctx, a.overhead gb print f"weights {a.file gb:.1f} GB + KV {kv:.1f} GB + overhead {a.overhead gb:.1f} GB = {need:.1f} GB" if need <= a.vram gb: print f"FITS in {a.vram gb:.0f} GB at ctx={a.ctx}. Max context about {max ctx}." else: print f"DOES NOT FIT in {a.vram gb:.0f} GB at ctx={a.ctx}. Max context about {max ctx}." if name == " main ": main Run it with your own values. The example below uses a hypothetical cost of 0.25 MiB per token to show the output format. It is not the measured cost of Qwen3.8-27B, so substitute your own figure. python fit.py --file-gb 16.5 --vram-gb 24 --kv-mib-per-token 0.25 --ctx 32768 weights 16.5 GB + KV 8.0 GB + overhead 1.5 GB = 26.0 GB DOES NOT FIT in 24 GB at ctx=32768. Max context about 24576. python fit.py --file-gb 16.5 --vram-gb 16 --kv-mib-per-token 0.25 --ctx 16384 weights 16.5 GB + KV 4.0 GB + overhead 1.5 GB = 22.0 GB DOES NOT FIT in 16 GB at ctx=16384. Max context about 0. The second result reads as a hard no. A max context of zero means the weights plus overhead already exceed the 16 GB budget. The answer there is partial offload, which this calculator does not model, or a lower-bit quant. When a context length does not fit, you have three levers. Cut the context, quantize the KV cache, or step down a quant. Quantizing the KV cache to 8-bit with -ctk q8 0 -ctv q8 0 roughly halves its size relative to 16-bit, at a small quality cost that you should test. Stepping from Q5 to Q4 frees a few gigabytes. Cutting context is free if you do not need it, and most chat sessions use far less than the maximum. Steps 9 and 10: Write the Streaming Client and the Benchmark Runner The harness has two jobs. The client sends one request and records exactly when the first token and the last token arrive. The runner decides which prompts to send, repeats them, and writes the rows to disk. Step 9: The streaming client Two metrics matter for interactive use. Time to first token TTFT captures how long you wait before anything appears, and it is dominated by prompt processing. Decode speed captures how fast text flows afterward, and it is dominated by memory bandwidth. A single tokens-per-second figure blends the two and hides the problem, which is why many casual benchmarks mislead. The client below reports both. python import json import os import time import urllib.request from dataclasses import dataclass @dataclass class Result: ttft s: float decode tps: float prompt tokens: int completion tokens: int total s: float text: str def post url, body, timeout : headers = {"Content-Type": "application/json"} if os.environ.get "LLAMA API KEY" : headers "Authorization" = "Bearer " + os.environ "LLAMA API KEY" req = urllib.request.Request url, data=json.dumps body .encode , headers=headers return urllib.request.urlopen req, timeout=timeout def count tokens base url, text, timeout=60 : with post base url.rstrip "/" + "/tokenize", {"content": text}, timeout as r: return len json.load r "tokens" def stream chat base url, model, prompt, max tokens=128, temperature=0.0, timeout=900, extra=None : body = {"model": model, "messages": {"role": "user", "content": prompt} , "max tokens": max tokens, "temperature": temperature, "stream": True, "stream options": {"include usage": True}} if extra: body.update extra t0 = time.perf counter t first = t last = None chunks, usage, parts = 0, {}, with post base url.rstrip "/" + "/v1/chat/completions", body, timeout as resp: for raw in resp: line = raw.decode "utf-8", "replace" .strip if not line.startswith "data:" : continue payload = line 5: .strip if payload == " DONE ": break evt = json.loads payload if evt.get "usage" : usage = evt "usage" for choice in evt.get "choices", : delta = choice.get "delta", {} piece = delta.get "content" or delta.get "reasoning content" or "" if piece: now = time.perf counter t first = t first or now t last = now chunks += 1 parts.append piece if t first is None: raise RuntimeError "server returned no tokens" done = usage.get "completion tokens", chunks span = t last - t first tps = done - 1 / span if done 1 and span 0 else 0.0 return Result t first - t0, tps, usage.get "prompt tokens", 0 , done, time.perf counter - t0, "".join parts A few design choices deserve a closer look. The client counts the first token from either content or reasoning content , so thinking models do not show a false TTFT of zero. Decode speed divides completion tokens minus one by the span between first and last token, because the first token arrives at the end of prefill and does not belong to the decode phase. The server’s own usage block supplies the token counts, and the code falls back to counting chunks when a server omits it. Finally, an API key in the LLAMA API KEY environment variable is attached as a bearer header, which matters once you secure the server in the advanced section. Step 10: The benchmark runner The runner builds prompts of controlled length, because prompt length is the variable that most changes speed. It asks the server’s tokenizer how long a candidate prompt is, then pads until it hits the target. Every prompt starts with a random run tag. That detail is not cosmetic. llama-server caches the processed prefix of previous prompts by default, so identical prompts after the first would skip prefill and report a flattering TTFT. The random tag breaks the prefix match and forces honest prefill work on every run. python import argparse import json import random import time import urllib.request from llama client import count tokens, stream chat FILLER = "The quarterly report covers storage costs, network latency, build times, " "and the number of incidents closed by the on-call team. " def make prompt base url, target tokens, nonce : text = FILLER while count tokens base url, text < target tokens: text += FILLER max 1, target tokens // 40 head = f" run {nonce} Summarize the text below in three short bullet points.\n\n" return head + text def wait healthy base url, tries=60 : for in range tries : try: with urllib.request.urlopen base url.rstrip "/" + "/health", timeout=5 as r: if json.load r .get "status" == "ok": return except Exception: pass time.sleep 2 raise SystemExit "server never became healthy" def main : p = argparse.ArgumentParser p.add argument "--base-url", default="http://127.0.0.1:8080" p.add argument "--model", default="qwen3.8-27b" p.add argument "--targets", type=int, nargs="+", default= 512, 4096, 16384 p.add argument "--repeats", type=int, default=3 p.add argument "--max-tokens", type=int, default=128 p.add argument "--label", default="q4 k m" p.add argument "--out", default="results.jsonl" a = p.parse args wait healthy a.base url stream chat a.base url, a.model, "Say hello.", max tokens=8 warm-up with open a.out, "a" as fh: for target in a.targets: for i in range a.repeats : nonce = random.randint 100 000 000, 999 999 999 prompt = make prompt a.base url, target, nonce r = stream chat a.base url, a.model, prompt, max tokens=a.max tokens row = {"label": a.label, "target": target, "run": i, "prompt tokens": r.prompt tokens, "completion tokens": r.completion tokens, "ttft s": round r.ttft s, 3 , "decode tps": round r.decode tps, 2 , "prefill tps": round r.prompt tokens / r.ttft s, 1 if r.ttft s else 0} fh.write json.dumps row + "\n" fh.flush print row if name == " main ": main The runner also sends a tiny warm-up request first, so one-time costs such as kernel compilation and memory allocation do not pollute the first measured row. It appends rows to a JSON Lines file, which means you can stop and resume, or run several quants into the same file and compare them with the label field. If you want to extend the same pattern to security testing, our prompt injection test lab https://shattered.io/prompt-injection-test-lab-llm-harness-2026/ uses the same idea of fixed prompts and recorded rows. Steps 11 and 12: Smoke-Test the Harness, Then Benchmark the Real Model Testing a benchmark against a 27B model is slow and expensive, so test it against a fake first. The mock server below speaks just enough of the protocol to exercise every code path: health, tokenize and a streaming chat response with a usage block. python import json import time from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer class H BaseHTTPRequestHandler : def json self, obj : data = json.dumps obj .encode self.send response 200 self.send header "Content-Type", "application/json" self.send header "Content-Length", str len data self.end headers self.wfile.write data def do GET self : self. json {"status": "ok"} def do POST self : body = json.loads self.rfile.read int self.headers "Content-Length" if self.path == "/tokenize": return self. json {"tokens": list range len body "content" .split } self.send response 200 self.send header "Content-Type", "text/event-stream" self.end headers time.sleep 0.2 for i in range body.get "max tokens", 16 : evt = {"choices": {"delta": {"content": f"tok{i} "}} } self.wfile.write f"data: {json.dumps evt }\n\n".encode self.wfile.flush time.sleep 0.01 usage = {"choices": , "usage": {"prompt tokens": 100, "completion tokens": body.get "max tokens", 16 }} self.wfile.write f"data: {json.dumps usage }\n\ndata: DONE \n\n".encode def log message self, a : pass ThreadingHTTPServer "127.0.0.1", 8099 , H .serve forever Start the mock in one terminal, then point the runner at it from another. The tiny targets keep the run short. python mock server.py & python bench.py --base-url http://127.0.0.1:8099 --targets 50 200 --repeats 3 --max-tokens 40 --label mock-q4 python report.py results.jsonl The runner prints one row per request as it goes, and the final command prints the summary table. These numbers come from the mock, which sleeps for a fixed interval, so they say nothing about Qwen3.8-27B. They only prove that the plumbing works. {'label': 'mock-q4', 'target': 50, 'run': 0, 'prompt tokens': 100, 'completion tokens': 40, 'ttft s': 0.201, 'decode tps': 96.98, 'prefill tps': 497.6} | quant | ctx target | runs | median TTFT s | worst TTFT s | prefill tok/s | decode tok/s | |---|---|---|---|---|---|---| | mock-q4 | 50 | 3 | 0.20 | 0.20 | 498 | 97.5 | | mock-q4 | 200 | 3 | 0.20 | 0.20 | 497 | 97.8 | The report script is short. It groups rows by label and context target, then prints medians, because medians shrug off the occasional outlier caused by a background process. It also prints the worst TTFT, which tells you how bad a slow request can get. python import json import statistics import sys from collections import defaultdict def main path="results.jsonl" : groups = defaultdict list for line in open path : row = json.loads line groups row "label" , row "target" .append row print "| quant | ctx target | runs | median TTFT s | worst TTFT s | prefill tok/s | decode tok/s |" print "|---|---|---|---|---|---|---|" for label, target , rows in sorted groups.items : ttft = r "ttft s" for r in rows print f"| {label} | {target} | {len rows } | {statistics.median ttft :.2f} | {max ttft :.2f} | " f"{statistics.median r 'prefill tps' for r in rows :.0f} | " f"{statistics.median r 'decode tps' for r in rows :.1f} |" if name == " main ": main sys.argv 1: Delete the mock results before the real run, otherwise the file mixes fake and real rows. Then benchmark the real server from Step 6. pkill -f mock server.py rm results.jsonl python bench.py --label q4 k m --targets 512 4096 8000 --repeats 3 --max-tokens 128 python report.py results.jsonl Keep the largest target under the context size you launched with, minus the 128 output tokens, or the server will reject or truncate the request. With the Step 6 launch at 8192, a target of 8000 leaves just enough room. For longer targets such as 16384, restart the server with a larger -c after checking the fit calculator. To compare quants, stop the server, launch it with a different GGUF, and rerun the benchmark with a new label such as q5 k m . Append to the same results file, and the report prints both side by side. Record the llama.cpp build number and the GPU model next to each result in a notes file, since the report script does not capture them. How to read the numbers Prefill throughput should be much higher than decode throughput, because prefill processes many tokens in parallel while decode produces them one at a time. TTFT grows with context length, and a steep rise between 4K and 8K usually means the KV cache or compute buffers pushed part of the work off the GPU. Decode speed should fall slowly as context grows. Compare your decode number to the bandwidth ceiling from Step 1. Landing at half the ceiling is normal, while landing at a tenth points to CPU offload or a build without GPU support. For orientation only, community benchmarks of this model on consumer GPUs are circulating online. Those figures are unverified, come from single testers, and depend on settings you cannot see, so use them to judge the order of magnitude and nothing more. Your own table is the number that counts. For how this model stacks up against rivals on quality rather than speed, see our DeepSeek vs Qwen vs Mistral comparison https://shattered.io/deepseek-vs-qwen-vs-mistral-open-weight-models-2026/ , and browse the AI and machine learning section https://shattered.io/category/ai-machine-learning/ for newer model coverage. Common Pitfalls That Skew Local LLM Benchmarks Most bad local benchmarks come from the same handful of mistakes. Check this list against your own setup before you trust a number. 1. Warm prompt cache. Sending the same prompt twice makes the second TTFT look instant. The runner’s random tag prevents it. If you write your own prompts, keep that habit. 2. Mixing GB and GiB. File sizes in listings may use either unit, and the gap is about 7%. Add the overhead cushion and leave margin rather than planning to the last gigabyte. 3. Context set too high. A launch context larger than the KV cache can hold either fails at startup or silently spills to system memory, which tanks speed. Measure first, then raise. 4. Thinking tokens counted as answers. In reasoning mode, the model emits many tokens before the visible answer. That inflates completion counts and hides latency. Decide whether you measure with reasoning on or off, and keep it fixed. 5. Comparing different quantizers. Two files both named Q4 K M from different uploaders can differ in calibration data and quality. Compare quants from one source when you judge quality. 6. Exposing the server. Binding to 0.0.0.0 without an API key hands your GPU to anyone on the network. Keep the loopback address by default. 7. Ignoring thermals and neighbors. A browser with hardware acceleration, a game or a second inference process steals memory bandwidth. Close them, and watch GPU clocks during long runs. None of these costs money to avoid, and each one has fooled experienced engineers. The pattern is the same every time: a number that looks great, a setup nobody can reproduce, and a surprise when the model goes into real use. Troubleshooting: 12 Problems and Fixes These are the failures readers hit most often with this stack, ordered roughly by when they appear. 1. Unknown model architecture at load. Your llama.cpp build is older than the model’s support. Pull the latest source, rebuild, and try again. 2. CUDA out of memory at startup. Lower -c , lower -ngl so a few layers stay on the CPU, quantize the KV cache, or choose a smaller quant. Run the fit calculator first next time. 3. Generation is painfully slow. Check the startup log for the offloaded layer count. A build compiled without a GPU backend, or -ngl 0 , runs everything on the CPU. 4. Garbled or repetitive output. The chat template is wrong. Launch with --jinja and make sure the GGUF embeds the template. Some uploads strip it. 5. A service-unavailable error right after launch. The model is still loading. The runner’s health check waits up to two minutes. Raise the retry count for slow disks. 6. Connection refused. The server crashed or you used the wrong port. Look at the server terminal first, and confirm the host and port flags match the client URL. 7. Empty content, populated reasoning content. The token budget ran out during thinking. Raise max tokens or lower the reasoning effort as the model card describes. 8. Unauthorized errors from the harness. You started the server with an API key. Export LLAMA API KEY in the shell that runs the client. 9. Tokens per second collapses at long context. The KV cache or compute buffers no longer fit on the GPU. Compare TTFT at each target and rerun the fit calculator for that length. 10. Hugging Face download stalls or returns 401. Rerun the same command to resume. If the repository is gated, accept its terms on the website and run hf auth login . 11. The harness reports server returned no tokens. The request produced only empty chunks, often because max tokens was too small for a reasoning model or the context was exceeded. Read the server log for the rejection reason. 12. Results differ wildly between runs. Look for background GPU load, thermal throttling or power-saving modes. Raise --repeats and compare the median, not the best run. When none of these match, reproduce the problem with the smallest possible command, copy the exact llama.cpp build number, and search the project’s issue tracker. Many failures with brand-new model families are already known and fixed in a newer build. Advanced Tips: Speculative Decoding, KV Quantization and Hardening Once the baseline numbers exist, you can test each optimization against them. Change one thing at a time and keep the label field honest. Speculative decoding. A small draft model proposes several tokens, and the large model verifies them in one pass. When the draft guesses well, decode speed rises without changing the output. The draft must share the main model’s tokenizer. Community testers have reportedly used a very low-bit quant of the same model as the draft, and multi-token prediction support has been moving through llama.cpp as well, though we could not confirm release details. Run llama-server --help for the current draft flags, such as -md for the draft model path, then benchmark with and without the draft using two labels. KV cache quantization. Add -ctk q8 0 -ctv q8 0 to shrink the cache and fit longer context. Rerun the Step 12 benchmark at your longest target and compare both TTFT and answer quality. Quantizing the key cache to lower bit widths costs more quality than the value cache, so test before you push further. Parallel slots. The -np flag splits the context across several simultaneous request slots. That helps a team share one server, but each slot gets only a fraction of the context. Benchmark with the concurrency you expect rather than a single stream. Vision input. The model includes a vision encoder, and in llama.cpp image input usually requires a separate multimodal projector file passed with --mmproj . Some community uploads bundle the file and some do not, so read the repository description. Harden the server. If anything beyond your own laptop will reach it, set --api-key , keep the bind on loopback and put a reverse proxy with TLS in front. A local model endpoint is an unauthenticated compute service until you say otherwise. For a long-running machine, wrap the launch command in a systemd unit so it restarts after a reboot. Unit Description=llama-server Qwen3.8-27B After=network.target Service User=llm WorkingDirectory=/home/llm/localbench ExecStart=/home/llm/llama.cpp/build/bin/llama-server -m models/Qwen3.8-27B-Q4 K M.gguf --alias qwen3.8-27b -c 8192 -ngl 99 --jinja --host 127.0.0.1 --port 8080 --api-key-file /etc/llama/api-key Restart=on-failure Install WantedBy=multi-user.target Check llama-server --help for the exact name of the API key option in your build, since builds differ on whether they read a key from a flag or a file. The unit above shows the shape, so adapt the flag to what your build accepts. Point real tools at it. Anything that speaks the OpenAI API can use your server by setting the base URL to http://127.0.0.1:8080/v1 and any non-empty key. Local coding assistants are the obvious fit, and our Aider local setup guide https://shattered.io/aider-local-ai-coding-assistant-ollama-git-2026/ shows the pattern with a different backend. If you later want to adapt the model to your own data, the QLoRA fine-tuning tutorial https://shattered.io/fine-tune-llm-qlora-12-steps-2026/ picks up from a base model like this one. Rerun the whole benchmark whenever you change the llama.cpp build, the quant or the driver. A saved results file per configuration turns hardware upgrades and software updates into decisions backed by data instead of impressions. FAQ: Running an LLM Locally With Qwen3.8-27B How much memory do I need to run an LLM locally? Add the GGUF file size, the KV cache for your context and about 1.5 GB of overhead. For a Q4-class Qwen3.8-27B file near 16.5 GB, that points to a 24 GB GPU or a machine with 32 GB or more of unified memory for comfortable use. Smaller budgets work with partial CPU offload at lower speed. Is Qwen3.8-27B the newest Qwen model? It is the newest Qwen model sized for a single workstation. The Qwen3.8 family also includes the 2.4-trillion-parameter Qwen3.8-2.4T-A95B and the hosted Qwen3.8-Max, and Alibaba says Qwen4 is in training but not released. Check the official Qwen pages before you start, since the line moves quickly. Can I run Qwen3.8-27B without a GPU? Yes. llama.cpp runs on the CPU, and the harness will report the speed honestly. Expect much lower decode speed than on a GPU because system memory has far less bandwidth. A smaller quant helps, and so does closing other programs. Why does the benchmark measure TTFT and decode speed separately? They have different bottlenecks. Prompt processing is compute-heavy and sets how long you wait for the first word. Token generation is memory-bandwidth-heavy and sets how fast the rest arrives. A single blended tokens-per-second number hides which one is holding you back. Do I need Python 3.15.2 specifically? No. The scripts use only the standard library and ran on Python 3.12.3 during testing. We recommend 3.15.2 because it is the newest stable release, and because Python 3.10 has reached end of life. Is it safe to download community GGUF files? GGUF is a data format, which is safer than executable formats, but treat files from unknown uploaders as untrusted. Prefer established quantizers, compare file sizes with the repository listing, and read the model card. Some uploads deliberately modify the model’s behavior. How do I know my result is trustworthy? Run at least three repeats, keep the random run tag so the prompt cache cannot help, record the llama.cpp build and the GPU, and compare the median against the bandwidth ceiling from Step 1. A number that beats the ceiling is a measurement bug, not a breakthrough. Can I use the same harness for other models? Yes. Launch any model behind an OpenAI-compatible server, pass its alias with --model , and label the rows. The client and runner make no assumptions about the model, which is the point of keeping them separate from the server setup.