{"slug": "2x-tesla-p100s-q6-k-quant-qwen-3-8-27b-60tps", "title": "2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps", "summary": "A fork of llama.cpp tuned for two Tesla P100 GPUs (sm_60) runs Qwen3.8-27B at 32.6 tokens per second in plain decode, 1.86x upstream's 17.5 t/s, and about 54 t/s with the model's built-in MTP head, roughly 3x upstream. Prefill roughly doubles from 250 to 493 t/s at short context, the full 262k window fits with vision loaded, and a question on a full 260k context prefills at about 153 t/s. The fork reports KLD against an fp32 run of 0.00119 versus 0.00152 for upstream's fp16 path, and tracks upstream by merging.", "body_md": "**This is a fork of llama.cpp tuned for two Tesla P100s (sm_60).** On Qwen3.8-27B, plain decode\nruns at 1.86x upstream's speed (17.5 -> 32.6 t/s), and ~3x with the model's built-in MTP head\n(~54 t/s). Prefill runs at ~2x (250 -> 493 t/s at short context). The full 262k window fits with vision\nloaded, and a question on a full 260k context prefills at ~153 t/s.\nThe math is at least as accurate as stock: KLD against an fp32 run 0.00119, where upstream's\nfp16 path reads 0.00152. It tracks upstream by merging.\nSee **[p100-docs/](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/p100-docs/README.md)** for results, changes and how to run it.\nUpstream's README follows.\n\n**LLM inference in C/C++**\n\n[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](<https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc>) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)\n\nA few options to get `llama.cpp` installed on your machine:\n\n- Visit [https://llama.app](https://llama.app) and follow the instructions\n- Run with Docker - see our [Docker documentation](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/docker.md)\n- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)\n- Build from source by cloning this repository - check out [our build guide](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md)\n\nOnce installed:\n\n```\n# Download and run a model directly from Hugging Face\nllama cli -hf ggml-org/Qwen3.5-0.8B-GGUF\n\n# Launch OpenAI-compatible API server\nllama serve -hf ggml-org/Qwen3.5-0.8B-GGUF\n```\n\n|  *VLM session with **llama cli*** |  *Built-in web UI against **llama serve*** | \n\nThe main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on\na wide range of hardware - locally and in the cloud.\n\n- Plain C/C++ implementation without any dependencies\n- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks\n- AVX, AVX2, AVX512 and AMX support for x86 architectures\n- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures\n- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use\n- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)\n- Vulkan and SYCL backend support\n- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity\n\nThe `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library.\n\n| Backend | Target devices | \n|---|---|\n| [BLAS](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#blas-build) | All | \n| [BLIS](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/BLIS.md) | All | \n| [CANN](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#cann) | Ascend NPU | \n| [CUDA](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#cuda) | Nvidia GPU | \n| [HIP](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#hip) | AMD GPU | \n| [Hexagon](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/snapdragon/README.md) | Snapdragon | \n| [IBM zDNN](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/zDNN.md) | IBM Z & LinuxONE | \n| [MUSA](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#musa) | Moore Threads GPU | \n| [Metal](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#metal-build) | Apple Silicon | \n| [OpenCL](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/OPENCL.md) | Adreno GPU | \n| [OpenVINO \\[In Progress\\]](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs | \n| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All | \n| [SYCL](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/SYCL.md) | Intel GPU | \n| [VirtGPU](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/VirtGPU.md) | VirtGPU APIR | \n| [Vulkan](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#vulkan) | GPU | \n| [WebGPU](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#webgpu) | All | \n| [ZenDNN](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#zendnn) | AMD CPU | \n\n- Contributors can open PRs\n- Collaborators will be invited based on contributions\n- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the`master` branch\n- Any help with managing issues, PRs and projects is very appreciated!\n- Read the [CONTRIBUTING.md](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/CONTRIBUTING.md) for more information\n\n- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by`llama-server` - MIT license\n- [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain\n- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License\n- [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain\n- [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain", "url": "https://wpnews.pro/news/2x-tesla-p100s-q6-k-quant-qwen-3-8-27b-60tps", "canonical_source": "https://github.com/Kmic-68/llama.cpp", "published_at": "2026-10-18 22:47:56+00:00", "updated_at": "2026-10-18 23:18:57.930774+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "machine-learning"], "entities": ["llama.cpp", "Tesla P100", "Qwen3.8-27B", "ggml", "NVIDIA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/2x-tesla-p100s-q6-k-quant-qwen-3-8-27b-60tps", "markdown": "https://wpnews.pro/news/2x-tesla-p100s-q6-k-quant-qwen-3-8-27b-60tps.md", "text": "https://wpnews.pro/news/2x-tesla-p100s-q6-k-quant-qwen-3-8-27b-60tps.txt", "jsonld": "https://wpnews.pro/news/2x-tesla-p100s-q6-k-quant-qwen-3-8-27b-60tps.jsonld"}}