Run an LLM Locally: Qwen3.8-27B in 12 Steps, 75 Min [2026]

Qwen3.8-27B is the member of the Qwen3.8 family that a single workstation can actually host. The Qwen model listing shows the dense 27-billion-parameter model, along with a matching FP8 release, dating to August 13, 2026, and the Qwen Cloud page for Qwen3.8-27B now sits next to community GGUF builds that run in llama.cpp, Ollama and LM Studio. If you want to run an LLM locally in late October 2026, this is the model to size your machine against.

Running it takes three pieces. You need weights that fit in memory, a server that speaks the OpenAI wire format, and a way to measure whether the result is fast enough to use. Most guides stop after the second piece and leave you guessing about the third. The steps below build all three in roughly 75 minutes, and most of that is download time. For a chat front end on top of the same server, our llama.cpp and Open WebUI walkthrough picks up where Step 6 ends.

Local inference is no longer a hobby corner. In the 2025 Stack Overflow Developer Survey, 14.4% of professional developers who use AI tools selected Ollama, and the survey’s technology section shows large language models among the tools more than a quarter of all respondents use or want to use. Numbers like that explain why people search for ways to run an LLM locally, and why a repeatable benchmark matters more than a screenshot of one lucky run.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

Why Qwen3.8-27B Is the Local Model to Start With #

Alibaba’s Qwen line moved to the Qwen3.8 generation over the summer. The family now spans a very large open flagship, Qwen3.8-2.4T-A95B, which carries 2.4 trillion total parameters with about 95 billion active per token, and a set of smaller releases meant for ordinary hardware. The 27B model is the one sized for a desktop. It is a dense causal language model with a vision encoder, so it reads images and video as well as text, and it offers an adjustable reasoning mode that you can switch on for hard problems and off for quick answers.

Dense matters for local use. A mixture-of-experts model activates only a slice of its weights per token, which helps speed, but you still need memory for all the experts. A dense 27B model has no such trick. Every weight is read for every generated token, so memory bandwidth sets your speed ceiling and file size sets your memory floor. That makes the arithmetic in this guide simple and predictable, which is exactly what you want when you are sizing a machine.

The official model repository documents Transformers, vLLM and SGLang as supported runtimes. llama.cpp support arrives through GGUF files that community members convert and upload. That split shapes the whole tutorial. The weights come from Qwen, the quantized files usually come from third parties, and you need a habit of checking one against the other. Step 5 covers how.

Two caveats on the rest of the family. Alibaba announced updates to Qwen3.8-Max at its Apsara Conference on October 28, 2026, including 33 automated improvement cycles and a rise in its Artificial Analysis score from 40 to 45, according to Philstar’s report on the conference. That is a hosted flagship, not something you run at home. We covered the open-weight side of that story in our Qwen3.8-Max open-weights analysis. Separately, Alibaba’s strategy update on October 2 said Qwen4 is in training. It has not shipped. One newsletter reported on October 31 that an Alibaba executive hinted at a Qwen4 release as soon as next week, but no release notice backs that up, so treat it as unconfirmed.

The practical takeaway: Qwen3.8-27B is the newest Qwen model that fits on one machine today, and the harness you build here will measure a Qwen4 or any other GGUF model just as easily when one arrives. Swap the model file, change the label, rerun.

Prerequisites and Tested Versions #

The harness uses only the Python standard library, so there is nothing to compile on the Python side. The heavy lifting happens in llama.cpp. The table lists what to have in place before you start, with the version guidance that applies on October 31, 2026.

Component	Version to use	Notes
Python	3.15.2 (released Oct. 26, 2026)	Python 3.10 reached end of life on Oct. 1, 2026. The scripts here also ran cleanly on 3.12.3 during testing.
llama.cpp	Newest build from ggml-org/llama.cpp	Builds carry a b-prefixed number. Older builds may not recognize the Qwen3.8 architecture.
Model	Qwen3.8-27B as a community GGUF	The official repository ships standard weights. Pick a quantizer you trust.
GPU drivers	Current CUDA 12.x, ROCm, or Metal	Vulkan also works when no vendor stack is available.
Hugging Face CLI	Latest huggingface_hub	Used for resumable downloads.
Disk space	40 GB free	Covers the model file, a source build and cache.
System RAM	32 GB recommended	Needed for partial offload and while the file loads.
Operating system	Linux, macOS or Windows 11	Commands below use a Bash shell. Windows users can use WSL 2 or PowerShell equivalents.

You do not need a GPU to follow along. A CPU-only run works, and the benchmark will simply report slower numbers. You do need enough memory to hold the weights, and Step 1 shows how to check that before you spend an hour on a download.

Step 1: Check Your Hardware and Set a Memory Budget #

Three things consume memory when a model runs: the weights, the key-value (KV) cache that grows with context length, and a fixed overhead for compute buffers and the GPU driver. Weights are the big, predictable part. Multiply the parameter count by the bits per weight, divide by eight, and you have the file size. At 27 billion parameters and an average near 4.9 bits per weight, that gives about 16.5 GB, which matches the Q4_K_M files that community listings describe.

Speed has its own rule of thumb. Because the model is dense, each generated token reads roughly the whole file from memory. Divide your memory bandwidth by the file size and you get a ceiling on decode speed. A card with 1,000 GB/s of bandwidth and a 16.5 GB file tops out near 60 tokens per second. Real runs land below that ceiling, so use it as a sanity check on your benchmark rather than a target. If your measured number beats the ceiling, something is wrong with the measurement.

Run the commands for your platform and write down the free memory figure. That number is your budget.

nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv

free -g

sysctl hw.memsize

The next table maps common memory budgets to a starting plan. It is analysis, not a vendor claim, and Step 8 replaces the guesswork with a measured calculation.

Memory available to the model	Starting plan	What to expect
Under 12 GB	Lower-bit quant with heavy CPU offload, or choose a smaller model	Slow decode. The 27B model is a stretch here.
12 to 16 GB	Q4-class quant with partial GPU offload	The weights alone strain the card, so some layers run on the CPU.
20 to 24 GB	Q4-class quant fully on the GPU, moderate context	Comfortable at 8K to 16K context. Longer context needs a KV check.
32 GB or more	Q5 or Q6 quant fully on the GPU	Room for long context or a second model for speculative decoding.
Unified memory, 32 GB and up	Q4 to Q6 quant, all layers offloaded	Bandwidth, not capacity, limits speed. Expect a lower ceiling than a flagship GPU.

Owners of a flagship consumer card have a head start, and our guide to running LLMs locally on an RTX 5090 covers that hardware in depth. AMD users have an official path too. According to AMD’s own blog post, Qwen3.8-27B is supported on Ryzen AI Max systems and Radeon graphics cards, and the AMD post lists the supported setup.

On Windows, remember that the display driver reserves some video memory, so a 24 GB card rarely offers a full 24 GB. On a laptop with a hybrid GPU, confirm that llama.cpp sees the discrete card and not the integrated one. These two details cause more first-run failures than any flag.

Steps 2 and 3: Install Python 3.15.2 and Create the Project Folder #

Install Python 3.15.2 from python.org, your distribution’s package manager, pyenv or any other installer you trust. Then confirm the version and create an isolated environment. The harness has no third-party dependencies, but the Hugging Face CLI does, and a virtual environment keeps it away from your system packages.

python3.15 --version

mkdir -p ~/localbench/models && cd ~/localbench
python3.15 -m venv .venv
source .venv/bin/activate            # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install -U pip huggingface_hub

If the first command prints an older number, your shell is finding a different interpreter. Call the binary by its full path or fix your PATH before continuing. Mixing interpreters is a quiet source of confusion later, especially when a script runs fine in one terminal and fails in another.

The finished project has five Python files and one models folder. You will create each file in its own step, so the folder starts nearly empty.

localbench/
  models/                  # GGUF files live here
  fit.py                   # Step 8: will it fit in memory?
  llama_client.py          # Step 9: streaming client with timing
  bench.py                 # Step 10: benchmark runner
  mock_server.py           # Step 11: fake server for a smoke test
  report.py                # Step 12: summary table
  results.jsonl            # written by bench.py

Keeping the pieces separate pays off. The client knows how to talk to a server and time it. The runner decides what to ask. The report reads rows and summarizes them. When you later point the same code at vLLM or Ollama, only the base URL changes.

Step 4: Install llama.cpp #

llama.cpp is an open-source C and C++ inference engine maintained under the ggml-org organization on GitHub. It has no paid tiers. Development is fast, with new builds landing almost daily. On October 30 the release feed reportedly showed pre-release builds in the b12000 range, so a build from a few weeks ago may lack support for newer model architectures. Build from the current source or grab the newest release binary.

Release binaries exist for macOS, Windows and Linux, including CUDA variants, and they are the fastest route. Building from source takes five to ten minutes and gives you the exact backend you want. The commands below build with CUDA. Swap the flag for your hardware.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_CUDA=ON

cmake --build build --config Release -j
./build/bin/llama-server --version

Expect a version line that includes a build number. Write it down, because every benchmark you run should record the build it came from. Performance shifts between builds, and a result without a build number is hard to reproduce a month later.

Some community documentation now shows a newer single-command form, llama serve and llama cli, while many repositories still call the standalone llama-server and llama-cli binaries. Both forms exist in the wild. The commands here use llama-server, so if your build only has the newer subcommands, run llama serve with the same flags.

Not sure llama.cpp is the right runtime for you? Our comparison of Ollama, LM Studio, llama.cpp and vLLM lays out the trade-offs. We use llama.cpp here for three reasons. Its flags are explicit, so nothing happens behind your back. Its server exposes a tokenizer endpoint the harness needs. And it exposes Prometheus metrics for longer experiments.

Step 5: Download a Qwen3.8-27B GGUF #

Start at the official source so you know what the model is supposed to be. The Qwen organization’s Hugging Face repository for Qwen3.8-27B documents the architecture, the reasoning options and the license, and a matching FP8 repository sits beside it. Read the model card before you download anything else. The license file is the authority on commercial use. One third-party report describes the model as Apache 2.0, but confirm that in the official repository yourself rather than relying on a summary, including this one.

The official repository serves standard weights. For llama.cpp you want a GGUF, and the community has published several. Quantizers such as bartowski and unsloth publish well-known sets, and the benchmark repositories we found used an unsloth Q4_K_M file of about 16.5 GB. Smaller and larger quants exist, along with variants that bundle extra features. For a first run, take the Q4_K_M file from a quantizer with a track record.

source .venv/bin/activate
hf download bartowski/Qwen3.8-27B-GGUF \
  --include "*Q4_K_M*" \
  --local-dir models

ls -lh models/

If the include pattern matches nothing, list the repository files in your browser and adjust the pattern, because ups name their files differently. The hf download command resumes interrupted transfers, which saves you when a 16 GB file stalls at 90%.

How to read quant names

A quant name tells you the bit width and the method. Q4_K_M uses a mixed four-bit scheme and is the usual balance of size and quality. Q5_K_M and Q6_K trade more memory for less quality loss. Q8_0 comes close to the original precision but nearly doubles the file. IQ-prefixed names, such as IQ4_NL, use importance-matrix techniques to squeeze quality out of fewer bits. One community listing puts an IQ4_NL build at about 15.2 GiB, and other uploads claim sizes near 14 GB. Treat those figures as claims until you see the file size on your own disk.

Quality differences between neighboring quants are small for most chat and coding work, and large differences show up at the low end, below four bits. If you are unsure, the harness gives you the means to test. Download two quants, benchmark both, and compare quality on your own prompts alongside speed.

Weights from unknown ups deserve suspicion. Some community variants alter the model’s behavior on purpose, for example by stripping safety training. That may be fine for a research box and wrong for a tool you hand to colleagues. Check who published the file, compare its size and hash against the repository listing, and keep the official model card open when you evaluate. Our open-weight model license comparison explains why license terms deserve the same attention as benchmark scores.

Step 6: Launch llama-server #

llama-server loads the model once and serves an OpenAI-compatible HTTP API, plus a small built-in web page. Start with a conservative context size and increase it after you have measured the KV cost in Step 8.

./llama.cpp/build/bin/llama-server \
  -m models/Qwen3.8-27B-Q4_K_M.gguf \
  --alias qwen3.8-27b \
  -c 8192 \
  -ngl 99 \
  -fa on \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  --metrics

Adjust the -m path to the exact filename you downloaded. On a machine without a GPU, drop -ngl 99. The table explains each flag.

Flag	What it does	When to change it
-m	Path to the GGUF file	Always set. Use the exact filename.
–alias	Model name the API reports	Match the name your client sends.
-c	Context window in tokens	Raise after measuring KV cache cost.
-ngl	Layers to offload to the GPU	Lower it if you run out of VRAM. 99 means offload everything.
-fa	Flash attention, on or off	Leave on where supported. Older builds took a bare flag, so check the help output.
–jinja	Use the model’s own chat template	Keep on so reasoning and tool-call formatting match the model card.
–host and –port	Network binding	Keep 127.0.0.1 unless you add an API key and a firewall rule.
–metrics	Expose Prometheus counters	Useful for long soak tests.

Flag syntax shifts between builds, so run llama-server --help when something is rejected. The flash-attention switch is the usual offender, since some builds take a bare -fa and newer ones take on, off or auto.

Watch the startup log. You want to see the model file open without an architecture error, a line stating how many layers moved to the GPU, and a final message that the server is listening. A message about an unknown architecture means your llama.cpp build predates the model, and the fix is a newer build. If the layer count is lower than the total, part of the model runs on the CPU, which explains slow generation later.

Machines with unified memory, such as Apple silicon and the DGX Spark class of boxes, handle this step differently, since the GPU and CPU share one pool. Our DGX Spark walkthrough covers the details for that hardware, and the flags above carry over.

Step 7: Test the API with curl #

Before any Python, prove the server answers. Open a second terminal and hit three endpoints: health, a chat completion and the tokenizer.

curl -s http://127.0.0.1:8080/health

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Reply with the single word: ready"}],"max_tokens":64}'

curl -s http://127.0.0.1:8080/tokenize \
  -H "Content-Type: application/json" \
  -d '{"content":"How many tokens is this sentence?"}'

The health call should return a small JSON object whose status is ok. While the model is still , it returns an error status instead, so wait and retry. The chat call returns the standard OpenAI structure with a choices array, a message and a usage block. The tokenizer call returns a list of integer token IDs.

{"status":"ok"}

{"choices":[{"message":{"role":"assistant","content":"ready", ...}}],
 "usage":{"prompt_tokens": ..., "completion_tokens": ..., "total_tokens": ...}, ...}

{"tokens":[ ... ]}

Those dots mark values that differ on every machine. If your chat response has an empty content field and a populated reasoning_content field, the model spent its whole token budget thinking. Raise max_tokens or turn the reasoning mode down. The model card names the exact switch, and earlier Qwen3 releases used a chat-template option called enable_thinking, so check whether the 3.8 card keeps that name before you rely on it.

This is also the moment to try the built-in web page at the server’s root address. It gives you a quick feel for the model’s tone and speed before you measure anything.

Step 8: Measure KV Cache Cost and Run the Fit Calculator #

The KV cache stores attention state for every token in the context, and its size depends on the model’s layer count, attention layout and the cache precision. Qwen has not published a per-token figure on any page we found, and guessing one would mislead you. Measure it instead. Restart the server with -c 8192 as in Step 6, find the startup log line that reports the KV cache size in MiB, and divide by 8192. The result is your MiB-per-token cost.

Now feed that number to the calculator. It adds the file size, the KV cache for your target context and a 1.5 GB overhead cushion, then tells you whether the total fits and how much context your leftover memory could hold.

import argparse

def estimate(file_gb, vram_gb, kv_mib_per_token, ctx, overhead_gb=1.5):
    kv_gb = kv_mib_per_token * ctx / 1024
    need_gb = file_gb + kv_gb + overhead_gb
    spare_gb = vram_gb - file_gb - overhead_gb
    max_ctx = int(spare_gb * 1024 / kv_mib_per_token) if spare_gb > 0 and kv_mib_per_token > 0 else 0
    return need_gb, kv_gb, max_ctx

def main():
    p = argparse.ArgumentParser(description="Will this GGUF fit in memory?")
    p.add_argument("--file-gb", type=float, required=True, help="size of the .gguf on disk")
    p.add_argument("--vram-gb", type=float, required=True, help="usable GPU or unified memory")
    p.add_argument("--kv-mib-per-token", type=float, required=True, help="measured from the server log")
    p.add_argument("--ctx", type=int, default=16384)
    p.add_argument("--overhead-gb", type=float, default=1.5)
    a = p.parse_args()
    need, kv, max_ctx = estimate(a.file_gb, a.vram_gb, a.kv_mib_per_token, a.ctx, a.overhead_gb)
    print(f"weights {a.file_gb:.1f} GB + KV {kv:.1f} GB + overhead {a.overhead_gb:.1f} GB = {need:.1f} GB")
    if need <= a.vram_gb:
        print(f"FITS in {a.vram_gb:.0f} GB at ctx={a.ctx}. Max context about {max_ctx}.")
    else:
        print(f"DOES NOT FIT in {a.vram_gb:.0f} GB at ctx={a.ctx}. Max context about {max_ctx}.")

if __name__ == "__main__":
    main()

Run it with your own values. The example below uses a hypothetical cost of 0.25 MiB per token to show the output format. It is not the measured cost of Qwen3.8-27B, so substitute your own figure.

python fit.py --file-gb 16.5 --vram-gb 24 --kv-mib-per-token 0.25 --ctx 32768
weights 16.5 GB + KV 8.0 GB + overhead 1.5 GB = 26.0 GB
DOES NOT FIT in 24 GB at ctx=32768. Max context about 24576.

python fit.py --file-gb 16.5 --vram-gb 16 --kv-mib-per-token 0.25 --ctx 16384
weights 16.5 GB + KV 4.0 GB + overhead 1.5 GB = 22.0 GB
DOES NOT FIT in 16 GB at ctx=16384. Max context about 0.

The second result reads as a hard no. A max context of zero means the weights plus overhead already exceed the 16 GB budget. The answer there is partial offload, which this calculator does not model, or a lower-bit quant.

When a context length does not fit, you have three levers. Cut the context, quantize the KV cache, or step down a quant. Quantizing the KV cache to 8-bit with -ctk q8_0 -ctv q8_0 roughly halves its size relative to 16-bit, at a small quality cost that you should test. Stepping from Q5 to Q4 frees a few gigabytes. Cutting context is free if you do not need it, and most chat sessions use far less than the maximum.

Steps 9 and 10: Write the Streaming Client and the Benchmark Runner #

The harness has two jobs. The client sends one request and records exactly when the first token and the last token arrive. The runner decides which prompts to send, repeats them, and writes the rows to disk.

Step 9: The streaming client

Two metrics matter for interactive use. Time to first token (TTFT) captures how long you wait before anything appears, and it is dominated by prompt processing. Decode speed captures how fast text flows afterward, and it is dominated by memory bandwidth. A single tokens-per-second figure blends the two and hides the problem, which is why many casual benchmarks mislead. The client below reports both.

import json
import os
import time
import urllib.request
from dataclasses import dataclass

@dataclass
class Result:
    ttft_s: float
    decode_tps: float
    prompt_tokens: int
    completion_tokens: int
    total_s: float
    text: str

def _post(url, body, timeout):
    headers = {"Content-Type": "application/json"}
    if os.environ.get("LLAMA_API_KEY"):
        headers["Authorization"] = "Bearer " + os.environ["LLAMA_API_KEY"]
    req = urllib.request.Request(url, data=json.dumps(body).encode(), headers=headers)
    return urllib.request.urlopen(req, timeout=timeout)

def count_tokens(base_url, text, timeout=60):
    with _post(base_url.rstrip("/") + "/tokenize", {"content": text}, timeout) as r:
        return len(json.load(r)["tokens"])

def stream_chat(base_url, model, prompt, max_tokens=128, temperature=0.0, timeout=900, extra=None):
    body = {"model": model, "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens, "temperature": temperature, "stream": True,
            "stream_options": {"include_usage": True}}
    if extra:
        body.update(extra)
    t0 = time.perf_counter()
    t_first = t_last = None
    chunks, usage, parts = 0, {}, []
    with _post(base_url.rstrip("/") + "/v1/chat/completions", body, timeout) as resp:
        for raw in resp:
            line = raw.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                break
            evt = json.loads(payload)
            if evt.get("usage"):
                usage = evt["usage"]
            for choice in evt.get("choices", []):
                delta = choice.get("delta", {})
                piece = delta.get("content") or delta.get("reasoning_content") or ""
                if piece:
                    now = time.perf_counter()
                    t_first = t_first or now
                    t_last = now
                    chunks += 1
                    parts.append(piece)
    if t_first is None:
        raise RuntimeError("server returned no tokens")
    done = usage.get("completion_tokens", chunks)
    span = t_last - t_first
    tps = (done - 1) / span if done > 1 and span > 0 else 0.0
    return Result(t_first - t0, tps, usage.get("prompt_tokens", 0), done,
                  time.perf_counter() - t0, "".join(parts))

A few design choices deserve a closer look. The client counts the first token from either content or reasoning_content, so thinking models do not show a false TTFT of zero. Decode speed divides completion tokens minus one by the span between first and last token, because the first token arrives at the end of prefill and does not belong to the decode phase. The server’s own usage block supplies the token counts, and the code falls back to counting chunks when a server omits it. Finally, an API key in the LLAMA_API_KEY environment variable is attached as a bearer header, which matters once you secure the server in the advanced section.

Step 10: The benchmark runner

The runner builds prompts of controlled length, because prompt length is the variable that most changes speed. It asks the server’s tokenizer how long a candidate prompt is, then pads until it hits the target. Every prompt starts with a random run tag. That detail is not cosmetic. llama-server caches the processed prefix of previous prompts by default, so identical prompts after the first would skip prefill and report a flattering TTFT. The random tag breaks the prefix match and forces honest prefill work on every run.

import argparse
import json
import random
import time
import urllib.request

from llama_client import count_tokens, stream_chat

FILLER = ("The quarterly report covers storage costs, network latency, build times, "
          "and the number of incidents closed by the on-call team. ")

def make_prompt(base_url, target_tokens, nonce):
    text = FILLER
    while count_tokens(base_url, text) < target_tokens:
        text += FILLER * max(1, target_tokens // 40)
    head = f"[run {nonce}] Summarize the text below in three short bullet points.\n\n"
    return head + text

def wait_healthy(base_url, tries=60):
    for _ in range(tries):
        try:
            with urllib.request.urlopen(base_url.rstrip("/") + "/health", timeout=5) as r:
                if json.load(r).get("status") == "ok":
                    return
        except Exception:
            pass
        time.sleep(2)
    raise SystemExit("server never became healthy")

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--base-url", default="http://127.0.0.1:8080")
    p.add_argument("--model", default="qwen3.8-27b")
    p.add_argument("--targets", type=int, nargs="+", default=[512, 4096, 16384])
    p.add_argument("--repeats", type=int, default=3)
    p.add_argument("--max-tokens", type=int, default=128)
    p.add_argument("--label", default="q4_k_m")
    p.add_argument("--out", default="results.jsonl")
    a = p.parse_args()

    wait_healthy(a.base_url)
    stream_chat(a.base_url, a.model, "Say hello.", max_tokens=8)  # warm-up
    with open(a.out, "a") as fh:
        for target in a.targets:
            for i in range(a.repeats):
                nonce = random.randint(100_000_000, 999_999_999)
                prompt = make_prompt(a.base_url, target, nonce)
                r = stream_chat(a.base_url, a.model, prompt, max_tokens=a.max_tokens)
                row = {"label": a.label, "target": target, "run": i,
                       "prompt_tokens": r.prompt_tokens, "completion_tokens": r.completion_tokens,
                       "ttft_s": round(r.ttft_s, 3), "decode_tps": round(r.decode_tps, 2),
                       "prefill_tps": round(r.prompt_tokens / r.ttft_s, 1) if r.ttft_s else 0}
                fh.write(json.dumps(row) + "\n")
                fh.flush()
                print(row)

if __name__ == "__main__":
    main()

The runner also sends a tiny warm-up request first, so one-time costs such as kernel compilation and memory allocation do not pollute the first measured row. It appends rows to a JSON Lines file, which means you can stop and resume, or run several quants into the same file and compare them with the label field. If you want to extend the same pattern to security testing, our prompt injection test lab uses the same idea of fixed prompts and recorded rows.

Steps 11 and 12: Smoke-Test the Harness, Then Benchmark the Real Model #

Testing a benchmark against a 27B model is slow and expensive, so test it against a fake first. The mock server below speaks just enough of the protocol to exercise every code path: health, tokenize and a streaming chat response with a usage block.

import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class H(BaseHTTPRequestHandler):
    def _json(self, obj):
        data = json.dumps(obj).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        self._json({"status": "ok"})

    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        if self.path == "/tokenize":
            return self._json({"tokens": list(range(len(body["content"].split())))})
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        time.sleep(0.2)
        for i in range(body.get("max_tokens", 16)):
            evt = {"choices": [{"delta": {"content": f"tok{i} "}}]}
            self.wfile.write(f"data: {json.dumps(evt)}\n\n".encode())
            self.wfile.flush()
            time.sleep(0.01)
        usage = {"choices": [], "usage": {"prompt_tokens": 100, "completion_tokens": body.get("max_tokens", 16)}}
        self.wfile.write(f"data: {json.dumps(usage)}\n\ndata: [DONE]\n\n".encode())

    def log_message(self, *a):
        pass

ThreadingHTTPServer(("127.0.0.1", 8099), H).serve_forever()

Start the mock in one terminal, then point the runner at it from another. The tiny targets keep the run short.

python mock_server.py &
python bench.py --base-url http://127.0.0.1:8099 --targets 50 200 --repeats 3 --max-tokens 40 --label mock-q4
python report.py results.jsonl

The runner prints one row per request as it goes, and the final command prints the summary table. These numbers come from the mock, which sleeps for a fixed interval, so they say nothing about Qwen3.8-27B. They only prove that the plumbing works.

{'label': 'mock-q4', 'target': 50, 'run': 0, 'prompt_tokens': 100, 'completion_tokens': 40, 'ttft_s': 0.201, 'decode_tps': 96.98, 'prefill_tps': 497.6}

| quant | ctx target | runs | median TTFT (s) | worst TTFT (s) | prefill tok/s | decode tok/s |
|---|---|---|---|---|---|---|
| mock-q4 | 50 | 3 | 0.20 | 0.20 | 498 | 97.5 |
| mock-q4 | 200 | 3 | 0.20 | 0.20 | 497 | 97.8 |

The report script is short. It groups rows by label and context target, then prints medians, because medians shrug off the occasional outlier caused by a background process. It also prints the worst TTFT, which tells you how bad a slow request can get.

import json
import statistics
import sys
from collections import defaultdict

def main(path="results.jsonl"):
    groups = defaultdict(list)
    for line in open(path):
        row = json.loads(line)
        groups[(row["label"], row["target"])].append(row)
    print("| quant | ctx target | runs | median TTFT (s) | worst TTFT (s) | prefill tok/s | decode tok/s |")
    print("|---|---|---|---|---|---|---|")
    for (label, target), rows in sorted(groups.items()):
        ttft = [r["ttft_s"] for r in rows]
        print(f"| {label} | {target} | {len(rows)} | {statistics.median(ttft):.2f} | {max(ttft):.2f} | "
              f"{statistics.median(r['prefill_tps'] for r in rows):.0f} | "
              f"{statistics.median(r['decode_tps'] for r in rows):.1f} |")

if __name__ == "__main__":
    main(*sys.argv[1:])

Delete the mock results before the real run, otherwise the file mixes fake and real rows. Then benchmark the real server from Step 6.

pkill -f mock_server.py
rm results.jsonl

python bench.py --label q4_k_m --targets 512 4096 8000 --repeats 3 --max-tokens 128
python report.py results.jsonl

Keep the largest target under the context size you launched with, minus the 128 output tokens, or the server will reject or truncate the request. With the Step 6 launch at 8192, a target of 8000 leaves just enough room. For longer targets such as 16384, restart the server with a larger -c after checking the fit calculator.

To compare quants, stop the server, launch it with a different GGUF, and rerun the benchmark with a new label such as q5_k_m. Append to the same results file, and the report prints both side by side. Record the llama.cpp build number and the GPU model next to each result in a notes file, since the report script does not capture them.

How to read the numbers

Prefill throughput should be much higher than decode throughput, because prefill processes many tokens in parallel while decode produces them one at a time. TTFT grows with context length, and a steep rise between 4K and 8K usually means the KV cache or compute buffers pushed part of the work off the GPU. Decode speed should fall slowly as context grows. Compare your decode number to the bandwidth ceiling from Step 1. Landing at half the ceiling is normal, while landing at a tenth points to CPU offload or a build without GPU support.

For orientation only, community benchmarks of this model on consumer GPUs are circulating online. Those figures are unverified, come from single testers, and depend on settings you cannot see, so use them to judge the order of magnitude and nothing more. Your own table is the number that counts. For how this model stacks up against rivals on quality rather than speed, see our DeepSeek vs Qwen vs Mistral comparison, and browse the AI and machine learning section for newer model coverage.

Common Pitfalls That Skew Local LLM Benchmarks #

Most bad local benchmarks come from the same handful of mistakes. Check this list against your own setup before you trust a number.

1. Warm prompt cache. Sending the same prompt twice makes the second TTFT look instant. The runner’s random tag prevents it. If you write your own prompts, keep that habit.
2. Mixing GB and GiB. File sizes in listings may use either unit, and the gap is about 7%. Add the overhead cushion and leave margin rather than planning to the last gigabyte.
3. Context set too high. A launch context larger than the KV cache can hold either fails at startup or silently spills to system memory, which tanks speed. Measure first, then raise.
4. Thinking tokens counted as answers. In reasoning mode, the model emits many tokens before the visible answer. That inflates completion counts and hides latency. Decide whether you measure with reasoning on or off, and keep it fixed.
5. Comparing different quantizers. Two files both named Q4_K_M from different ups can differ in calibration data and quality. Compare quants from one source when you judge quality.
6. Exposing the server. Binding to 0.0.0.0 without an API key hands your GPU to anyone on the network. Keep the loopback address by default.
7. Ignoring thermals and neighbors. A browser with hardware acceleration, a game or a second inference process steals memory bandwidth. Close them, and watch GPU clocks during long runs.

None of these costs money to avoid, and each one has fooled experienced engineers. The pattern is the same every time: a number that looks great, a setup nobody can reproduce, and a surprise when the model goes into real use.

Troubleshooting: 12 Problems and Fixes #

These are the failures readers hit most often with this stack, ordered roughly by when they appear.

1. Unknown model architecture at load. Your llama.cpp build is older than the model’s support. Pull the latest source, rebuild, and try again.
2. CUDA out of memory at startup. Lower-c , lower-ngl so a few layers stay on the CPU, quantize the KV cache, or choose a smaller quant. Run the fit calculator first next time.
3. Generation is painfully slow. Check the startup log for the offloaded layer count. A build compiled without a GPU backend, or-ngl 0 , runs everything on the CPU.
4. Garbled or repetitive output. The chat template is wrong. Launch with--jinja and make sure the GGUF embeds the template. Some uploads strip it.
5. A service-unavailable error right after launch. The model is still . The runner’s health check waits up to two minutes. Raise the retry count for slow disks.
6. Connection refused. The server crashed or you used the wrong port. Look at the server terminal first, and confirm the host and port flags match the client URL.
7. Empty content, populated reasoning_content. The token budget ran out during thinking. Raisemax_tokens or lower the reasoning effort as the model card describes.
8. Unauthorized errors from the harness. You started the server with an API key. ExportLLAMA_API_KEY in the shell that runs the client.
9. Tokens per second collapses at long context. The KV cache or compute buffers no longer fit on the GPU. Compare TTFT at each target and rerun the fit calculator for that length.
10. Hugging Face download stalls or returns 401. Rerun the same command to resume. If the repository is gated, accept its terms on the website and runhf auth login .
11. The harness reports server returned no tokens. The request produced only empty chunks, often becausemax_tokens was too small for a reasoning model or the context was exceeded. Read the server log for the rejection reason.
12. Results differ wildly between runs. Look for background GPU load, thermal throttling or power-saving modes. Raise--repeats and compare the median, not the best run.

When none of these match, reproduce the problem with the smallest possible command, copy the exact llama.cpp build number, and search the project’s issue tracker. Many failures with brand-new model families are already known and fixed in a newer build.

Advanced Tips: Speculative Decoding, KV Quantization and Hardening #

Once the baseline numbers exist, you can test each optimization against them. Change one thing at a time and keep the label field honest.

Speculative decoding. A small draft model proposes several tokens, and the large model verifies them in one pass. When the draft guesses well, decode speed rises without changing the output. The draft must share the main model’s tokenizer. Community testers have reportedly used a very low-bit quant of the same model as the draft, and multi-token prediction support has been moving through llama.cpp as well, though we could not confirm release details. Run llama-server --help for the current draft flags, such as -md for the draft model path, then benchmark with and without the draft using two labels.

KV cache quantization. Add -ctk q8_0 -ctv q8_0 to shrink the cache and fit longer context. Rerun the Step 12 benchmark at your longest target and compare both TTFT and answer quality. Quantizing the key cache to lower bit widths costs more quality than the value cache, so test before you push further.

Parallel slots. The -np flag splits the context across several simultaneous request slots. That helps a team share one server, but each slot gets only a fraction of the context. Benchmark with the concurrency you expect rather than a single stream.

Vision input. The model includes a vision encoder, and in llama.cpp image input usually requires a separate multimodal projector file passed with --mmproj. Some community uploads bundle the file and some do not, so read the repository description.

Harden the server. If anything beyond your own laptop will reach it, set --api-key, keep the bind on loopback and put a reverse proxy with TLS in front. A local model endpoint is an unauthenticated compute service until you say otherwise. For a long-running machine, wrap the launch command in a systemd unit so it restarts after a reboot.

[Unit]
Description=llama-server Qwen3.8-27B
After=network.target

[Service]
User=llm
WorkingDirectory=/home/llm/localbench
ExecStart=/home/llm/llama.cpp/build/bin/llama-server -m models/Qwen3.8-27B-Q4_K_M.gguf --alias qwen3.8-27b -c 8192 -ngl 99 --jinja --host 127.0.0.1 --port 8080 --api-key-file /etc/llama/api-key
Restart=on-failure

[Install]
WantedBy=multi-user.target

Check llama-server --help for the exact name of the API key option in your build, since builds differ on whether they read a key from a flag or a file. The unit above shows the shape, so adapt the flag to what your build accepts.

Point real tools at it. Anything that speaks the OpenAI API can use your server by setting the base URL to http://127.0.0.1:8080/v1 and any non-empty key. Local coding assistants are the obvious fit, and our Aider local setup guide shows the pattern with a different backend. If you later want to adapt the model to your own data, the QLoRA fine-tuning tutorial picks up from a base model like this one.

Rerun the whole benchmark whenever you change the llama.cpp build, the quant or the driver. A saved results file per configuration turns hardware upgrades and software updates into decisions backed by data instead of impressions.

FAQ: Running an LLM Locally With Qwen3.8-27B #

How much memory do I need to run an LLM locally?

Add the GGUF file size, the KV cache for your context and about 1.5 GB of overhead. For a Q4-class Qwen3.8-27B file near 16.5 GB, that points to a 24 GB GPU or a machine with 32 GB or more of unified memory for comfortable use. Smaller budgets work with partial CPU offload at lower speed.

Is Qwen3.8-27B the newest Qwen model?

It is the newest Qwen model sized for a single workstation. The Qwen3.8 family also includes the 2.4-trillion-parameter Qwen3.8-2.4T-A95B and the hosted Qwen3.8-Max, and Alibaba says Qwen4 is in training but not released. Check the official Qwen pages before you start, since the line moves quickly.

Can I run Qwen3.8-27B without a GPU?

Yes. llama.cpp runs on the CPU, and the harness will report the speed honestly. Expect much lower decode speed than on a GPU because system memory has far less bandwidth. A smaller quant helps, and so does closing other programs.

Why does the benchmark measure TTFT and decode speed separately?

They have different bottlenecks. Prompt processing is compute-heavy and sets how long you wait for the first word. Token generation is memory-bandwidth-heavy and sets how fast the rest arrives. A single blended tokens-per-second number hides which one is holding you back.

Do I need Python 3.15.2 specifically?

No. The scripts use only the standard library and ran on Python 3.12.3 during testing. We recommend 3.15.2 because it is the newest stable release, and because Python 3.10 has reached end of life.

Is it safe to download community GGUF files?

GGUF is a data format, which is safer than executable formats, but treat files from unknown ups as untrusted. Prefer established quantizers, compare file sizes with the repository listing, and read the model card. Some uploads deliberately modify the model’s behavior.

How do I know my result is trustworthy?

Run at least three repeats, keep the random run tag so the prompt cache cannot help, record the llama.cpp build and the GPU, and compare the median against the bandwidth ceiling from Step 1. A number that beats the ceiling is a measurement bug, not a breakthrough.

Can I use the same harness for other models?

Yes. Launch any model behind an OpenAI-compatible server, pass its alias with --model, and label the rows. The client and runner make no assumptions about the model, which is the point of keeping them separate from the server setup.

source & further reading

shattered.io — original article AI Code Security: Opus 5.5, GPT-6-Sol Tie at 73% [2026] Samaya Sterling Scores 58.9% vs Claude’s 55.9% [2026] Agent Compute Engine vs Substrate: 10x Claim Checked [2026]