2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps

A fork of llama.cpp tuned for two Tesla P100 GPUs (sm_60) runs Qwen3.8-27B at 32.6 tokens per second in plain decode, 1.86x upstream's 17.5 t/s, and about 54 t/s with the model's built-in MTP head, roughly 3x upstream. Prefill roughly doubles from 250 to 493 t/s at short context, the full 262k window fits with vision loaded, and a question on a full 260k context prefills at about 153 t/s. The fork reports KLD against an fp32 run of 0.00119 versus 0.00152 for upstream's fp16 path, and tracks upstream by merging.

This is a fork of llama.cpp tuned for two Tesla P100s sm 60 . On Qwen3.8-27B, plain decode runs at 1.86x upstream's speed 17.5 - 32.6 t/s , and ~3x with the model's built-in MTP head ~54 t/s . Prefill runs at ~2x 250 - 493 t/s at short context . The full 262k window fits with vision loaded, and a question on a full 260k context prefills at ~153 t/s. The math is at least as accurate as stock: KLD against an fp32 run 0.00119, where upstream's fp16 path reads 0.00152. It tracks upstream by merging. See p100-docs/ https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/p100-docs/README.md for results, changes and how to run it. Upstream's README follows. LLM inference in C/C++ ggml https://github.com/ggml-org/ggml / ops https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md / maintainer PRs <https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20 author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming %20sort%3Aupdated-desc / dev stats https://github.com/ggml-org/llama.cpp-dev / lib llama API https://github.com/ggml-org/llama.cpp/issues/9289 / llama-server REST API https://github.com/ggml-org/llama.cpp/issues/9291 A few options to get llama.cpp installed on your machine: - Visit https://llama.app https://llama.app and follow the instructions - Run with Docker - see our Docker documentation https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/docker.md - Download pre-built binaries from the releases page https://github.com/ggml-org/llama.cpp/releases - Build from source by cloning this repository - check out our build guide https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md Once installed: Download and run a model directly from Hugging Face llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF Launch OpenAI-compatible API server llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF | VLM session with llama cli | Built-in web UI against llama serve | The main goal of llama.cpp is to enable LLM and VLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud. - Plain C/C++ implementation without any dependencies - Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks - AVX, AVX2, AVX512 and AMX support for x86 architectures - RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures - 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use - Custom CUDA kernels for running LLMs on NVIDIA GPUs support for AMD GPUs via HIP and Moore Threads GPUs via MUSA - Vulkan and SYCL backend support - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity The llama.cpp project is build on top of the ggml https://github.com/ggml-org/ggml library. | Backend | Target devices | |---|---| | BLAS https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md blas-build | All | | BLIS https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/BLIS.md | All | | CANN https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md cann | Ascend NPU | | CUDA https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md cuda | Nvidia GPU | | HIP https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md hip | AMD GPU | | Hexagon https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/snapdragon/README.md | Snapdragon | | IBM zDNN https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/zDNN.md | IBM Z & LinuxONE | | MUSA https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md musa | Moore Threads GPU | | Metal https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md metal-build | Apple Silicon | | OpenCL https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/OPENCL.md | Adreno GPU | | OpenVINO \ In Progress\ https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/OPENVINO.md | Intel CPUs, GPUs, and NPUs | | RPC https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc | All | | SYCL https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/SYCL.md | Intel GPU | | VirtGPU https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/VirtGPU.md | VirtGPU APIR | | Vulkan https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md vulkan | GPU | | WebGPU https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md webgpu | All | | ZenDNN https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md zendnn | AMD CPU | - Contributors can open PRs - Collaborators will be invited based on contributions - Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch - Any help with managing issues, PRs and projects is very appreciated - Read the CONTRIBUTING.md https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/CONTRIBUTING.md for more information - yhirose/cpp-httplib https://github.com/yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license - nothings/stb https://github.com/nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain - nlohmann/json https://github.com/nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License - mackron/miniaudio https://github.com/mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain - sheredom/subprocess.h https://github.com/sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain