{"slug": "what-should-i-change-to-optimize-local-hosted-ai", "title": "What should i change to optimize local hosted AI", "summary": "A community analysis of local AI coding agent performance identifies several optimization settings for Ubuntu users running llama.cpp/SYCL with dual Intel Arc Pro B70 GPUs and the Qwen3.6-27B model. Key recommendations include adding the --perf flag for detailed logging, testing single-GPU mode before dual-GPU layer split, reducing CPU threads to 8 with --threads-batch 16, and comparing quantizations Q4_K_M vs Q5_K_M to improve latency.", "body_md": "Hmm… Based on known community findings, there do seem to be several settings worth improving:\n\nYour setup is not obviously wrong — Ubuntu + llama.cpp/SYCL + 2× Arc Pro B70 + Qwen3.6-27B is a reasonable direction for local coding agents. However, Intel Arc/Battlemage + llama.cpp/SYCL + Qwen3.6 has several known performance traps, so I would first build a clean baseline and then change one variable at a time.\n\nThe most important distinction is:\n\n**prefill / prompt processing**: shown as `prompt eval time`\n\n; very important for coding agents because they repeatedly send repo context, diffs, tool results, logs, etc.\n**decode / generation**: shown as `eval time`\n\n; important for normal token-by-token response speed\n**stability / memory behavior**: especially important with multi-GPU SYCL, MTP, large context, and long-running agents\n\nRelevant upstream docs/issues:\n\nTL;DR: highest-value things I would test first\n\n| Area |\nCurrent-looking setting |\nWhat I would test |\nWhy |\n| Measurement |\nno explicit perf logging |\nadd `--perf` |\nWithout `prompt eval` vs `eval` , tuning is guesswork |\n| GPU split |\n`--split-mode layer --tensor-split 1,1` |\ntry `--split-mode none --main-gpu 0` |\n27B Q5 may fit on one B70; dual-GPU split may add latency |\n| Threads |\n`--threads 24` |\ntry `--threads 8 --threads-batch 16` |\nGPU-offloaded inference often does not benefit from many CPU threads |\n| NUMA |\n`--numa distribute` |\nremove it first |\nlikely not useful on a normal single-socket workstation |\n| KV cache |\n`q8_0/q8_0` |\ncompare `f16` , `q8_0` , `q4_0` |\nArc/B70 quant paths can behave very differently |\n| Quant |\nQ5_K_M only |\ncompare Q4_K_M vs Q5_K_M |\nQ4 may be much better latency on B70 |\n| Flash Attention |\nunspecified/auto |\ntest `-fa on` vs `auto` |\noften relevant for long-context workloads |\n| Build |\nmoving target? |\npin/compare builds |\nknown `server-intel-b9159` regression exists |\n| Continue |\none big model for everything? |\nsplit autocomplete to smaller model |\nautocomplete should be latency-optimized |\n| MTP |\ntempting |\nleave it for later |\nQwen3.6 MTP + SYCL still has sharp edges |\n\n1. Add `--perf`\n\nbefore changing anything\n\nFirst, keep your current command but add:\n\n```\n--perf\n```\n\nThen look for:\n\n```\nprompt eval time\neval time\ntokens per second\n```\n\nInterpretation:\n\n- slow\n`prompt eval`\n\n= context/prefill problem\n- slow\n`eval`\n\n= generation/quant/backend/split problem\n- slow in Continue but not in\n`llama-bench`\n\n= likely agentic-context or client-side request pattern problem\n\nFor coding agents, `prompt eval`\n\nis often the hidden bottleneck. A model can look fine on short prompts or `tg128`\n\n, but feel bad in Continue because every agent step re-sends large context.\n\n2. Test single GPU before dual-GPU layer split\n\nYour current-style setup appears to use:\n\n```\n--split-mode layer \\\n--tensor-split 1,1\n```\n\nI would absolutely compare that with single-GPU mode:\n\n```\n--split-mode none \\\n--main-gpu 0\n```\n\nOptionally also pin the SYCL device:\n\n```\nexport ONEAPI_DEVICE_SELECTOR=level_zero:0\n```\n\nWhy this may help:\n\n- a 27B Q5_K_M model may fit on a single 32 GB B70\n- layer split helps capacity, but does not guarantee better single-user latency\n- decode often does not scale well across multiple GPUs\n- multi-GPU SYCL may increase host-memory pressure\n- Intel multi-GPU SYCL has a known host-side GTT mirror behavior:\n[SYCL multi-GPU GTT mirror issue](https://github.com/ggml-org/llama.cpp/issues/22116)\n\nIf single GPU is faster or similarly fast, I would use the second B70 for another service instead:\n\n```\nGPU 0: Qwen3.6-27B for chat/edit/agent\nGPU 1: Qwen2.5-Coder 1.5B/7B for autocomplete, or embeddings/reranking\n```\n\nFor a single-user coding workstation, two independent services can feel better than one model split over two GPUs.\n\n3. Reduce CPU threads and set batch threads separately\n\nIf you currently use:\n\n```\n--threads 24\n```\n\nI would compare:\n\n```\n--threads 8 \\\n--threads-batch 16\n```\n\nand:\n\n```\n--threads 4 \\\n--threads-batch 16\n```\n\nand:\n\n```\n--threads 8 \\\n--threads-batch 8\n```\n\n`--threads`\n\nand `--threads-batch`\n\nare separate llama.cpp server knobs. `--threads`\n\nis more relevant to generation-side CPU work, while `--threads-batch`\n\nmatters for batch/prompt processing. See the official [server option docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).\n\nWith most layers on GPU, more CPU threads are not always better. Too many threads can add scheduling overhead or just not help. For coding agents, `--threads-batch`\n\ncan matter more because large prompt ingestion is common.\n\n4. Remove `--numa distribute`\n\nunless this is really a NUMA machine\n\nIf this is a normal single-socket desktop/workstation system, I would remove:\n\n```\n--numa distribute\n```\n\nBaseline should probably be no NUMA setting. Only test NUMA modes later if you know the machine is actually NUMA-relevant.\n\n5. Do not assume `q8_0`\n\nKV cache is fastest\n\nYour command uses:\n\n```\n--cache-type-k q8_0 \\\n--cache-type-v q8_0\n```\n\nThat may be good for VRAM, but it should be measured. Compare:\n\n```\n--cache-type-k f16 \\\n--cache-type-v f16\n--cache-type-k q8_0 \\\n--cache-type-v q8_0\n--cache-type-k q4_0 \\\n--cache-type-v q4_0\n```\n\nThe B70-specific reason to test this is that quantized paths on Battlemage can behave surprisingly. The clearest known example is [Q8_0 being ~4× slower than Q4_K_M on Arc Pro B70](https://github.com/ggml-org/llama.cpp/issues/21517). That issue is about model weights, not KV cache, so it does **not** prove `q8_0`\n\nKV is bad. But it does prove that on B70, “higher bit = safer/faster” is not a reliable assumption.\n\nThe same issue also notes that `-DGGML_SYCL_F16=ON`\n\nimproved prompt processing by about 2.4× in one Q4_K_M case, while not improving token generation. That is another clue that prefill and decode must be tuned separately.\n\n6. Test Q4_K_M against Q5_K_M\n\nQ5_K_M is reasonable, but for local coding latency I would compare:\n\n```\nQwen3.6-27B-Q4_K_M\nQwen3.6-27B-Q5_K_M\nQwen3.6-27B-Q6_K\n```\n\nSuggested order:\n\n- Q4_K_M baseline\n- Q5_K_M quality comparison\n- Q6_K only if you still have enough speed/VRAM\n\nOn B70, Q4_K_M may be a better practical latency/quality point than Q5_K_M. The B70 Q8_0 issue is the strongest warning that quant performance on this architecture is not always intuitive: [B70 Q8_0 kernel efficiency issue](https://github.com/ggml-org/llama.cpp/issues/21517).\n\n7. Explicitly test Flash Attention\n\nTry:\n\n```\n-fa on\n```\n\nand compare with:\n\n```\n-fa auto\n```\n\nand maybe:\n\n```\n-fa off\n```\n\nFor long-context coding workloads, Flash Attention can matter, but it should still be measured. The option is documented in the [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).\n\n8. Prefer `--n-gpu-layers all`\n\nover `999`\n\nIf the intention is “offload everything possible,” use:\n\n```\n--n-gpu-layers all\n```\n\ninstead of:\n\n```\n--n-gpu-layers 999\n```\n\nThis is mostly clarity, not a guaranteed performance change. The server docs support `auto`\n\n, `all`\n\n, and numeric values.\n\n9. Pin builds; avoid moving `latest`\n\nThere is a very relevant Intel build regression report:\n\nThat issue is Qwen3.6-35B-A3B-MTP on Arc Pro B50, not exactly your 27B dense setup, so it is not proof that your setup is affected. But it is close enough to justify build pinning and A/B testing.\n\nRecord:\n\n```\n./build/bin/llama-server --version\nsycl-ls\nuname -a\n```\n\nIf using Docker, compare pinned images rather than `latest`\n\n:\n\n```\nghcr.io/ggml-org/llama.cpp:server-intel-b9144\nghcr.io/ggml-org/llama.cpp:server-intel-b9159\n```\n\nWith Intel Arc + SYCL, performance can depend on:\n\n- llama.cpp commit\n- Intel compute-runtime\n- oneAPI version\n- Linux kernel / driver stack\n- Docker image contents\n- whether\n`i915`\n\nor `xe`\n\nis used\n- ReBAR / Above 4G / PCIe platform behavior\n\n10. Check platform basics: ReBAR, PCIe, driver stack\n\nIf performance is much lower than other B70 reports, I would verify platform-level things too.\n\nUseful checks:\n\n```\nlspci -vv | grep -i -E \"Resizable BAR|Region|prefetchable\" -A3\nlspci -nnk | grep -i -E \"VGA|Display|3D\" -A4\nsycl-ls\nuname -a\n```\n\nThings to confirm:\n\n```\nAbove 4G Decoding: enabled\nResizable BAR: enabled\nPCIe link width/speed: expected width/speed\ndriver stack: i915 vs xe\nIntel compute-runtime version\noneAPI version\nkernel version\n```\n\nThere is also a relevant report of very poor SYCL performance on older DDR4 / PCIe 3.0 platform with Battlemage: [brutally bad SYCL performance on Battlemage](https://github.com/ggml-org/llama.cpp/issues/22413). Your system sounds much newer, but it is still worth verifying ReBAR/PCIe/driver basics.\n\n11. Treat MTP as a later experiment, not the first fix\n\nQwen3.6 MTP is interesting, but I would not add it until the non-MTP baseline is clean.\n\nRelevant issues:\n\nIf you test MTP later, start conservatively:\n\n```\n--parallel 1 \\\n--spec-type draft-mtp \\\n--spec-draft-n-max 2\n```\n\nAvoid assuming MTP is faster just because draft acceptance is high. On SYCL/Intel Arc, [one issue](https://github.com/ggml-org/llama.cpp/issues/23533) specifically reports correct output but no speed gain, with per-kernel dispatch overhead identified as the remaining bottleneck.\n\nAlso watch for:\n\n```\ndraft acceptance rate\nVRAM before/after requests\nforcing full prompt re-processing\ncreate context checkpoint\nOOM after multiple requests\n```\n\n12. Watch for full prompt re-processing\n\nFor Qwen3.6 and agentic use, look for:\n\n```\nforcing full prompt re-processing\n```\n\nRelevant issue:\n\nIf this appears often, the problem may not be raw GPU throughput. It may be prompt cache invalidation, slot reuse behavior, hybrid attention/recurrent-memory behavior, client request shape, or MTP interaction.\n\nFor a single local coding-agent user, I would initially force:\n\n```\n--parallel 1\n```\n\nand only increase parallelism after the baseline is stable.\n\n13. Continue.dev: separate autocomplete from chat/agent\n\nContinue has different model roles: Chat, Edit, Apply, Autocomplete, Embedding, Reranker, etc. See [Continue model roles](https://docs.continue.dev/customize/models).\n\nFor autocomplete specifically, Continue recommends smaller/faster models such as Qwen Coder 2.5 1.5B or 7B: [Continue autocomplete docs](https://docs.continue.dev/customize/deep-dives/autocomplete). The docs also note that thinking-type models are generally not recommended for autocomplete because they generate more slowly.\n\nA practical split:\n\n``` php\nlocalhost:8080 -> Qwen3.6-27B-Q4_K_M or Q5_K_M for chat/edit/agent\nlocalhost:8081 -> Qwen2.5-Coder-1.5B or 7B for autocomplete\n```\n\nThis can improve perceived responsiveness a lot. Autocomplete should not be queued behind a large 27B agent request.\n\n14. Suggested clean baseline\n\nI would start with something like this:\n\n``` bash\n#!/bin/bash\nsource /opt/intel/oneapi/setvars.sh --force\n\nexport ZES_ENABLE_SYSMAN=1\nexport ONEAPI_DEVICE_SELECTOR=level_zero:0\nexport UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1\n\ncd ~/llama.cpp\n\n./build/bin/llama-server \\\n  -m ~/models/Qwen3.6-27B-Q4_K_M.gguf \\\n  -a Roboto \\\n  -c 32768 \\\n  -fa on \\\n  --cache-type-k f16 \\\n  --cache-type-v f16 \\\n  --n-gpu-layers all \\\n  -b 2048 \\\n  -ub 512 \\\n  --threads 8 \\\n  --threads-batch 16 \\\n  --host 0.0.0.0 \\\n  --port 8080 \\\n  --split-mode none \\\n  --main-gpu 0 \\\n  --parallel 1 \\\n  --perf\n```\n\nThis is not guaranteed to be best. It is just a cleaner baseline:\n\n- one GPU\n- no NUMA complication\n- explicit Flash Attention\n- explicit KV type\n- explicit\n`threads`\n\nvs `threads-batch`\n\n- explicit\n`parallel 1`\n\n- perf logging\n- Q4_K_M as a latency-first starting point\n\nThen change only one thing at a time.\n\n15. Suggested A/B order\n\nStep 0: current setup + perf\n\nAdd only:\n\n```\n--perf\n```\n\nSave the logs.\n\nStep 1: single GPU\n\nChange:\n\n```\n--split-mode layer \\\n--tensor-split 1,1\n```\n\nto:\n\n```\n--split-mode none \\\n--main-gpu 0\n```\n\nIf this is faster, dual-GPU split is probably not helping latency.\n\nStep 2: threads\n\nTry:\n\n```\n--threads 8 \\\n--threads-batch 16\n```\n\ninstead of:\n\n```\n--threads 24\n```\n\nStep 3: remove NUMA\n\nRemove:\n\n```\n--numa distribute\n```\n\nStep 4: KV cache\n\nCompare:\n\n```\n--cache-type-k f16 --cache-type-v f16\n--cache-type-k q8_0 --cache-type-v q8_0\n--cache-type-k q4_0 --cache-type-v q4_0\n```\n\nStep 5: quant\n\nCompare:\n\n```\nQwen3.6-27B-Q4_K_M\nQwen3.6-27B-Q5_K_M\n```\n\nStep 6: Flash Attention\n\nCompare:\n\n```\n-fa on\n-fa auto\n-fa off\n```\n\nStep 7: context size\n\nCompare:\n\n```\n-c 16384\n-c 24576\n-c 32768\n```\n\nBigger context is not free. For coding agents, having 32K available is useful, but repeatedly filling it can make the system feel slow.\n\nStep 8: pinned builds\n\nCompare known builds / commits, especially if using Docker or recent server-intel images.\n\nStep 9: only then try MTP\n\nOnly after non-MTP is stable, try:\n\n```\n--parallel 1 \\\n--spec-type draft-mtp \\\n--spec-draft-n-max 2\n```\n\nand compare it against non-MTP.\n\n16. What I would put in the log comparison table\n\nSomething like this:\n\n```\nConfig name:\nModel:\nQuant:\nBackend:\nllama.cpp version:\nIntel compute-runtime:\noneAPI:\nKernel:\nDriver stack:\nGPU split:\nKV type:\nContext size:\nBatch / ubatch:\nThreads / threads-batch:\nFlash Attention:\nParallel slots:\n\nprompt eval t/s:\neval t/s:\nVRAM used:\nSystem RAM:\nGTT mirror, if checked:\nNotes:\n```\n\nFor multi-GPU SYCL memory behavior, this may help:\n\n```\nPID=$(pgrep llama-server)\n\nfor fd in /proc/$PID/fdinfo/*; do\n  grep -H \"drm-total-gtt\\|drm-total-vram\" \"$fd\" 2>/dev/null\ndone\n```\n\n17. My best guess\n\nMy guess is that the biggest practical wins will come from:\n\n**single-GPU baseline instead of dual-GPU layer split**\n**lower CPU thread count with explicit **`--threads-batch`\n\n**removing **`--numa distribute`\n\n**testing Q4_K_M vs Q5_K_M**\n**testing KV **`f16`\n\nvs `q8_0`\n\n**pinning known-good llama.cpp / server-intel builds**\n**separating Continue autocomplete onto a smaller model**\n\nI would not start by chasing MTP, huge context sizes, or experimental split modes. First make the normal Qwen3.6-27B path fast and reproducible.", "url": "https://wpnews.pro/news/what-should-i-change-to-optimize-local-hosted-ai", "canonical_source": "https://discuss.huggingface.co/t/what-should-i-change-to-optimize-local-hosted-ai/176339#post_3", "published_at": "2026-07-23 18:21:07+00:00", "updated_at": "2026-07-23 18:35:51.826753+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Ubuntu", "llama.cpp", "SYCL", "Intel Arc Pro B70", "Qwen3.6-27B", "Intel", "Qwen2.5-Coder"], "alternates": {"html": "https://wpnews.pro/news/what-should-i-change-to-optimize-local-hosted-ai", "markdown": "https://wpnews.pro/news/what-should-i-change-to-optimize-local-hosted-ai.md", "text": "https://wpnews.pro/news/what-should-i-change-to-optimize-local-hosted-ai.txt", "jsonld": "https://wpnews.pro/news/what-should-i-change-to-optimize-local-hosted-ai.jsonld"}}