# 2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps > Source: > Published: 2026-10-18 22:47:56+00:00 **This is a fork of llama.cpp tuned for two Tesla P100s (sm_60).** On Qwen3.8-27B, plain decode runs at 1.86x upstream's speed (17.5 -> 32.6 t/s), and ~3x with the model's built-in MTP head (~54 t/s). Prefill runs at ~2x (250 -> 493 t/s at short context). The full 262k window fits with vision loaded, and a question on a full 260k context prefills at ~153 t/s. The math is at least as accurate as stock: KLD against an fp32 run 0.00119, where upstream's fp16 path reads 0.00152. It tracks upstream by merging. See **[p100-docs/](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/p100-docs/README.md)** for results, changes and how to run it. Upstream's README follows. **LLM inference in C/C++** [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs]() / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291) A few options to get `llama.cpp` installed on your machine: - Visit [https://llama.app](https://llama.app) and follow the instructions - Run with Docker - see our [Docker documentation](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/docker.md) - Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases) - Build from source by cloning this repository - check out [our build guide](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md) Once installed: ``` # Download and run a model directly from Hugging Face llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF # Launch OpenAI-compatible API server llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF ``` | *VLM session with **llama cli*** | *Built-in web UI against **llama serve*** | The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud. - Plain C/C++ implementation without any dependencies - Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks - AVX, AVX2, AVX512 and AMX support for x86 architectures - RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures - 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use - Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA) - Vulkan and SYCL backend support - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library. | Backend | Target devices | |---|---| | [BLAS](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#blas-build) | All | | [BLIS](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/BLIS.md) | All | | [CANN](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#cann) | Ascend NPU | | [CUDA](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#cuda) | Nvidia GPU | | [HIP](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#hip) | AMD GPU | | [Hexagon](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/snapdragon/README.md) | Snapdragon | | [IBM zDNN](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/zDNN.md) | IBM Z & LinuxONE | | [MUSA](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#musa) | Moore Threads GPU | | [Metal](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#metal-build) | Apple Silicon | | [OpenCL](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/OPENCL.md) | Adreno GPU | | [OpenVINO \[In Progress\]](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs | | [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All | | [SYCL](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/SYCL.md) | Intel GPU | | [VirtGPU](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/backend/VirtGPU.md) | VirtGPU APIR | | [Vulkan](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#vulkan) | GPU | | [WebGPU](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#webgpu) | All | | [ZenDNN](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/docs/build.md#zendnn) | AMD CPU | - Contributors can open PRs - Collaborators will be invited based on contributions - Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the`master` branch - Any help with managing issues, PRs and projects is very appreciated! - Read the [CONTRIBUTING.md](https://github.com/Kmic-68/llama.cpp/blob/p100-optimizations/CONTRIBUTING.md) for more information - [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by`llama-server` - MIT license - [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain - [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License - [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain - [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain