KoboldCPP: Run GGUF Models Easily with a KoboldAI UI. One File. Zero Install

KoboldCpp, a single-file executable for running GGUF and GGML AI models, has been released by developer LostRuins. The tool requires no installation and supports text generation, image generation, speech recognition, and more, running on CPU or GPU across Windows, MacOS, and Linux. It builds on llama.cpp and provides a KoboldAI-inspired UI with multiple modes and API endpoints.

KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models, inspired by the original KoboldAI . It's a single self-contained distributable that builds off llama.cpp and adds many additional powerful features. Download Releases Here https://github.com/LostRuins/koboldcpp/releases/latest . - Single file executable, with no installation required and no external dependencies - Runs on CPU or GPU, supports full or partial offloaded - LLM text generation Supports all GGML and GGUF models, backwards compatibility with ALL past models - Image Generation and Image Editing Stable Diffusion 1.5, SDXL, SD3, Flux, Qwen Image, Z-Image, Klein - Video Generation WAN 2.2 - Speech-To-Text Voice Recognition via Whisper - Text-To-Speech Voice Generation via Qwen3TTS, Kokoro, OuteTTS, Parler and Dia - Music Generation Ace Step 1.5 - Image Recognition Multimodal Vision - MCP Server support and tool calling - Provides many compatible APIs endpoints for many popular webservices KoboldCppApi OpenAiApi OllamaApi A1111ForgeApi ComfyUiApi WhisperTranscribeApi XttsApi OpenAiSpeechApi - Bundled KoboldAI Lite UI with editing tools, save formats, memory, world info, author's note, characters, scenarios. - Includes multiple modes chat, adventure, instruct, storywriter and UI Themes aesthetic roleplay, classic writer, corporate assistant, messsenger - Supports loading Tavern Character Cards, importing many different data formats from various sites, reading or exporting JSON savefiles and persistent stories. - Many other features including new samplers, regex support, websearch, RAG via TextDB, image recognition/vision and more. - Ready-to-use binaries for Windows, MacOS, Linux. Runs directly with Colab, Docker, also supports other platforms if self-compiled like Android via Termux and Raspberry PI . Need help finding a model? Read this https://github.com/LostRuins/koboldcpp/wiki getting-an-ai-model-file - Windows binaries are provided in the form of koboldcpp.exe , which is a pyinstaller wrapper containing all necessary files. Download the latest koboldcpp.exe release here https://github.com/LostRuins/koboldcpp/releases/latest - To run, simply execute koboldcpp.exe . - Launching with no command line arguments displays a GUI containing a subset of configurable settings. Generally you dont have to change much besides the Presets and GPU Layers . Read the --help for more info about each settings. - Obtain and load a GGUF model. See here Obtaining-a-GGUF-model - By default, you can connect to http://localhost:5001 http://localhost:5001 - You can also run it using the command line. For info, please check koboldcpp.exe --help On modern Linux systems, you should download the koboldcpp-linux-x64 prebuilt PyInstaller binary on the releases page . Simply download and run the binary You may have to chmod +x it first . If you have an older device, you can also try the koboldcpp-linux-x64-oldpc instead for greatest compatibility.Alternatively, you can also install koboldcpp to the current directory by running the following terminal command: curl -fLo koboldcpp https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64-oldpc && chmod +x koboldcpp After running this command you can launch Koboldcpp from the current directory using ./koboldcpp in the terminal for CLI usage, run with --help . Finally, obtain and load a GGUF model. See here Obtaining-a-GGUF-model - PyInstaller binaries for Modern ARM64 MacOS M1, M2, M3 are now available Simply download the MacOS binary https://github.com/LostRuins/koboldcpp/releases/latest - In a MacOS terminal window, set the file to executable chmod +x koboldcpp-mac-arm64 and run it with ./koboldcpp-mac-arm64 . - In newer MacOS you may also have to whitelist it in security settings if it's blocked. Here's a video guide https://youtube.com/watch?v=NOW5dyA JgY . - Alternatively, or for older x86 MacOS computers, you can clone the repo and compile from source code, see Compiling for MacOS below. - Finally, obtain and load a GGUF model. See here Obtaining-a-GGUF-model - KoboldCpp now has an official Colab GPU Notebook This is an easy way to get started without installing anything in a minute or two. Try it here https://colab.research.google.com/github/LostRuins/koboldcpp/blob/concedo/colab.ipynb . - Note that KoboldCpp is not responsible for your usage of this Colab Notebook, you should ensure that your own usage complies with Google Colab's terms of use. - KoboldCpp can now be used on RunPod cloud GPUs This is an easy way to get started without installing anything in a minute or two, and is very scalable, capable of running 70B+ models at afforable cost. Try our RunPod image here https://koboldai.org/runpodcpp . Alternatively, you can also try SimplePod https://koboldai.org/simplepod for smaller models - Caution: The KoboldCpp docker is intended for experts only, and primarily intended for cloud GPU rental users If you're NOT an experienced user, you're recommended to use the precompiled binaries directly instead https://github.com/LostRuins/koboldcpp/releases/latest - The docker uses a x86-64 Ubuntu Linux based environment interally, and expects a Nvidia or AMD GPU. It may perform suboptimally on some Windows and MacOS devices, and may outright fail for ARM. It applies crude AVX/AVX2 feature detection which may not work correctly on all systems, resulting in the failsafe binaries being loaded speed will become extremely slow . - If you still want to proceed, the official docker can be found at https://hub.docker.com/r/koboldai/koboldcpp https://hub.docker.com/r/koboldai/koboldcpp - KoboldCpp uses GGUF models. They are not included with KoboldCpp, but you can download GGUF files from other places such as Bartowski's Huggingface https://huggingface.co/bartowski . Search for "GGUF" on huggingface.co for plenty of compatible models in the .gguf format. - For beginners, we recommend Qwen3-VL-8B https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF/resolve/main/Qwen3-VL-8B-Instruct-Q4 K S.gguf Most Recommended, best all rounder model - For creative writing and roleplay, you can try L3-8B-Stheno-v3.2 https://huggingface.co/bartowski/L3-8B-Stheno-v3.2-GGUF/resolve/main/L3-8B-Stheno-v3.2-Q4 K S.gguf old, smaller and weaker or Tiefighter 13B https://huggingface.co/KoboldAI/LLaMA2-13B-Tiefighter-GGUF/resolve/main/LLaMA2-13B-Tiefighter.Q4 K S.gguf old but very versatile model . Alternatively, you can download the tools to convert models to the GGUF format yourself here https://kcpptools.concedo.workers.dev . Run convert-hf-to-gguf.py to convert them, then quantize gguf.exe to quantize the result.- Other models for Whisper speech recognition , Image Generation, Text to Speech or Image Recognition can be found on the Wiki https://github.com/LostRuins/koboldcpp/wiki what-models-does-koboldcpp-support-what-architectures-are-supported GPU Acceleration : If you're on Windows with an Nvidia GPU you can get CUDA support out of the box using the --usecuda flag Nvidia Only , or --usevulkan Any GPU , make sure you select the correct .exe with CUDA support. GPU Layer Offloading : Add --gpulayers to offload model layers to the GPU. The more layers you offload to VRAM, the faster generation speed will become. Experiment to determine number of layers to offload, and reduce by a few if you run out of memory. Increasing Context Size : Use --contextsize number to increase context size, allowing the model to read more text. Note that you may also need to increase the max context in the KoboldAI Lite UI as well click and edit the number text field . Old CPU Compatibility : If you are having crashes or issues, you can try running in a non-avx2 compatibility mode by adding the --noavx2 flag. You can also try reducing your --blasbatchssize set -1 to avoid batching For more information, be sure to run the program with the --help flag, or check the wiki https://github.com/LostRuins/koboldcpp/wiki . when you can't use the precompiled binary directly, we provide an automated build script which uses conda to obtain all dependencies, and generates from source a ready-to-use a pyinstaller binary for linux users. - Clone the repo with git clone https://github.com/LostRuins/koboldcpp.git - Simply execute the build script with ./koboldcpp.sh dist and run the generated binary. Not recommended for systems that already have an existing installation of conda. Dependencies: curl, bzip2 ./koboldcpp.sh This launches the GUI for easy configuration and launching X11 required . ./koboldcpp.sh --help List all available terminal commands for using Koboldcpp, you can use koboldcpp.sh the same way as our python script and binaries. ./koboldcpp.sh rebuild Automatically generates a new conda runtime and compiles a fresh copy of the libraries. Do this after updating Koboldcpp to keep everything functional. ./koboldcpp.sh dist Generate your own precompiled binary Due to the nature of Linux compiling these will only work on distributions equal or newer than your own. - To compile your binaries from source, clone the repo with git clone https://github.com/LostRuins/koboldcpp.git - A makefile is provided, simply run make when compiling, you can set the number of parallel jobs with the -j flag . - Optional Vulkan: Link your own install of Vulkan SDK manually with make LLAMA VULKAN=1 - You can attempt a CuBLAS build with LLAMA CUBLAS=1 , or LLAMA HIPBLAS=1 for AMD . You will need CUDA Toolkit installed. Some have also reported success with the CMake file, though that is more for windows. - For a full featured build all backends , do make LLAMA CUBLAS=1 LLAMA VULKAN=1 . Note that LLAMA CUBLAS=1 will not work on windows, you need visual studio - To make your build sharable and capable of working on other devices, you must use LLAMA PORTABLE=1 - After all binaries are built, you can run the python script with the command python koboldcpp.py ggml model.gguf port - You're encouraged to use the .exe released, but if you want to compile your binaries from source at Windows, the easiest way is: - Get the latest release of w64devkit https://github.com/skeeto/w64devkit https://github.com/skeeto/w64devkit . Be sure to use the "vanilla one", not i686 or other different stuff. If you try they will conflit with the precompiled libs - Clone the repo with git clone https://github.com/LostRuins/koboldcpp.git - Make sure you are using the w64devkit integrated terminal, then run make at the KoboldCpp source folder. This will create the .dll files for a pure CPU native build when compiling, you can set the number of parallel jobs with the -j flag . - For a GPU build all backends , do make LLAMA VULKAN=1 . Note that LLAMA CUBLAS=1 will not work on windows, you need visual studio - To make your build sharable and capable of working on other devices, you must use LLAMA PORTABLE=1 - If you want to generate the .exe file, make sure you have the python module PyInstaller installed with pip pip install PyInstaller . Then run the script make pyinstaller.bat - The koboldcpp.exe file will be at your dist folder. - Get the latest release of w64devkit Building with CUDA : Visual Studio, CMake and CUDA Toolkit is required. Clone the repo, then open the CMake file and compile it in Visual Studio. Copy the koboldcpp cublas.dll generated into the same directory as the koboldcpp.py file. If you are bundling executables, you may need to include CUDA dynamic libraries such as cublasLt64 11.dll and cublas64 11.dll in order for the executable to work correctly on a different PC. Replacing Libraries Not Recommended : If you wish to use your own version of the additional Windows libraries Vulkan , you can do it with:- Move the respectives .lib files to the /lib folder of your project, overwriting the older files. - Also, replace the existing versions of the corresponding .dll files located in the project directory root. - Make the KoboldCpp project using the instructions above. - You can compile your binaries from source. You can clone the repo with git clone https://github.com/LostRuins/koboldcpp.git - A makefile is provided, simply run make when compiling, you can set the number of parallel jobs with the -j flag . - If you want Metal GPU support, instead run make LLAMA METAL=1 , note that MacOS metal libraries need to be installed. - To make your build sharable and capable of working on other devices, you must use LLAMA PORTABLE=1 - After all binaries are built, you can run the python script with the command python koboldcpp.py --model ggml model.gguf and add --gpulayers number of layer if you wish to offload layers to GPU . - Clone the repo with git clone https://github.com/LostRuins/koboldcpp.git - the project uses Gnu Makefile format, so you will need gmake: pkg add gmake - compiling vulkan support - you will require libvulkan, this is included in the vulkan-loader package, which is a dependency of the vulkan-tools package: pkg add vulkan-tools or pkg add vulkan-loader - you will require glslc, this is incliuded in the shaderc package: pkg add shaderc - if your gmake terminates with "fatal error: 'ggml-vulkan-shaders.hpp' file not found" the problem is probably that glslc is not installed. See above. - OpenBSD's default datasize limit may prevent compiliation ulimit -d 8388608 should work - compile using gmake LLAMA VULKAN=1 - you will require libvulkan, this is included in the vulkan-loader package, which is a dependency of the vulkan-tools package: - After all binaries are built, you can run the python script with the command python3 koboldcpp.py --model ggml model.gguf - You can use this auto-installation script to quickly install and build everything and launch KoboldCpp with a model. Simply run: curl -sSL https://raw.githubusercontent.com/LostRuins/koboldcpp/concedo/android install.sh | sh and it will install everything required. Alternatively, you can download the above android install.sh script to file, then do chmod +x and run it interactively. - Open termux and run the command apt update - Install dependency apt install openssl - Install other dependencies with pkg install wget git python - Run pkg upgrade - Clone the repo git clone https://github.com/LostRuins/koboldcpp.git - Navigate to the koboldcpp folder cd koboldcpp - Build the project make - To make your build sharable and capable of working on other devices, you must use LLAMA PORTABLE=1 , this disables usage of ARM instrinsics. - Grab a small GGUF model, such as wget https://huggingface.co/concedo/KobbleTinyV2-1.1B-GGUF/resolve/main/KobbleTiny-Q4 K.gguf - Start the python server python koboldcpp.py --model KobbleTiny-Q4 K.gguf - Connect to http://localhost:5001 on your mobile browser - If you encounter any errors, make sure your packages are up-to-date with pkg up and pkg upgrade - If you have trouble installing an dependency, you can try the command termux-change-repo and choose a different repo e.g. Mirror by BFSU - GPU acceleration for Termux may be possible but I have not explored it. If you find a good cross-device solution, do share or PR it. - For most users, you can get very decent speeds by selecting the Vulkan option instead, which supports both Nvidia and AMD GPUs. - Alternatively, you can try the ROCM fork at https://github.com/YellowRoseCx/koboldcpp-rocm https://github.com/YellowRoseCx/koboldcpp-rocm though this may be outdated. - These unofficial resources have been contributed by the community, and may be outdated or unmaintained. No official support will be provided for them - Arch Linux Packages: CUBLAS https://aur.archlinux.org/packages/koboldcpp-cuda , and HIPBLAS https://aur.archlinux.org/packages/koboldcpp-hipblas . - Unofficial Dockers: korewaChino https://github.com/korewaChino/koboldCppDocker and noneabove1182 https://github.com/noneabove1182/koboldcpp-docker - Nix & NixOS: KoboldCpp is available on Nixpkgs and can be installed by adding just koboldcpp to your environment.systemPackages or it can also be placed in . home.packages Example Nix Setup and further information /LostRuins/koboldcpp/blob/concedo/examples/nix example.md - If you face any issues with running KoboldCpp on Nix, please open an issue here https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug report.md&title= . - Arch Linux Packages: GPTLocalhost https://gptlocalhost.com/demo KoboldCpp - KoboldCpp is supported by GPTLocalhost, a local Word Add-in for you to use KoboldCpp in Microsoft Word. A local alternative to "Copilot in Word." First, please check out The KoboldCpp FAQ and Knowledgebase https://github.com/LostRuins/koboldcpp/wiki which may already have answers to your questions Also please search through past issues and discussions.- If you cannot find an answer, open an issue on this github, or find us on the KoboldAI Discord https://koboldai.org/discord . - For Windows: No installation, single file executable, It Just Works - Since v1.15, requires CLBlast if enabled, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without CLBlast. - Since v1.33, you can set the context size to be above what the model supports officially. It does increases perplexity but should still work well below 4096 even on untuned models. For GPT-NeoX, GPT-J, and Llama models Customize this with --ropeconfig . - Since v1.42, supports GGUF models for LLAMA and Falcon - Since v1.55, lcuda paths on Linux are hardcoded and may require manual changes to the makefile if you do not use koboldcpp.sh for the compilation. - Since v1.60, provides native image generation with StableDiffusion.cpp, you can load any SD1.5 or SDXL .safetensors model and it will provide an A1111 compatible API to use. I try to keep backwards compatibility with ALL past llama.cpp models . But you are also encouraged to reconvert/update your models if possible for best results.- Since v1.75, openblas has been deprecated and removed in favor of the native CPU implementation. - Since v1.107, CLBlast has been deprecated and removed in favor of Vulkan. - Phishing SCAM Warning: koboldcpp dot com is NOT an official site, please help to report it to google for impersonation. You should ONLY trust official downloads from the release binaries on the official github at https://github.com/LostRuins/koboldcpp/releases/latest https://github.com/LostRuins/koboldcpp/releases/latest - The original GGML library, stable-diffusion.cpp and llama.cpp by ggerganov are licensed under the MIT License - However, KoboldAI Lite is licensed under the AGPL v3.0 License - KoboldCpp code and other files are also under the AGPL v3.0 License unless otherwise stated - Llama.cpp source repo is at https://github.com/ggml-org/llama.cpp https://github.com/ggml-org/llama.cpp MIT - Stable-diffusion.cpp source repo is at https://github.com/leejet/stable-diffusion.cpp https://github.com/leejet/stable-diffusion.cpp MIT - TTS.cpp source repo is at https://github.com/mmwillet/TTS.cpp https://github.com/mmwillet/TTS.cpp MIT - Qwen3TTS source repo is at https://github.com/predict-woo/qwen3-tts.cpp https://github.com/predict-woo/qwen3-tts.cpp MIT - AceStep.cpp source repo is at https://github.com/ServeurpersoCom/acestep.cpp https://github.com/ServeurpersoCom/acestep.cpp MIT - KoboldCpp source repo is at https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp AGPL - KoboldAI Lite source repo is at https://github.com/LostRuins/lite.koboldai.net https://github.com/LostRuins/lite.koboldai.net AGPL - For any further enquiries, contact @concedo on discord, or LostRuins on github. - If you wish, after building the koboldcpp libraries with make , you can rebuild the exe yourself with pyinstaller by using make pyinstaller.bat - API documentation available at /api e.g. http://localhost:5001/api and https://lite.koboldai.net/koboldcpp api https://lite.koboldai.net/koboldcpp api . An OpenAI compatible API is also provided at /v1 route e.g. http://localhost:5001/v1 . All up-to-date GGUF models are supported , and KoboldCpp also includes backward compatibility for older versions/legacy GGML .bin models, though some newer features might be unavailable.- An incomplete list of architectures is listed, but there are many hundreds of other GGUF models . In general, if it's GGUF, it should work. - Llama / Llama2 / Llama3 / Alpaca / GPT4All / Vicuna / Koala / Pygmalion / Metharme / WizardLM / Mistral / Mixtral / Miqu / Qwen / Qwen2 / Yi / Gemma / Gemma2 / GPT-2 / Cerebras / Phi-2 / Phi-3 / GPT-NeoX / Pythia / StableLM / Dolly / RedPajama / GPT-J / RWKV4 / MPT / Falcon / Starcoder / Deepseek and many, many more. - The best place to get GGUF text models is huggingface. For image models, CivitAI has a good selection. Here are some to get started. - Text Generation: L3-8B-Stheno-v3.2 https://huggingface.co/bartowski/L3-8B-Stheno-v3.2-GGUF/resolve/main/L3-8B-Stheno-v3.2-Q4 K S.gguf smaller and weaker or Tiefighter 13B https://huggingface.co/KoboldAI/LLaMA2-13B-Tiefighter-GGUF/resolve/main/LLaMA2-13B-Tiefighter.Q4 K S.gguf old but very versatile model or Gemma-3-27B Abliterated https://huggingface.co/mlabonne/gemma-3-27b-it-abliterated-GGUF/resolve/main/gemma-3-27b-it-abliterated.q4 k m.gguf largest and most powerful - Image Generation: Anything v3 https://huggingface.co/admruul/anything-v3.0/resolve/main/Anything-V3.0-pruned-fp16.safetensors or Deliberate V2 https://huggingface.co/Yntec/Deliberate2/resolve/main/Deliberate v2.safetensors or Dreamshaper SDXL https://huggingface.co/Lykon/dreamshaper-xl-v2-turbo/resolve/main/DreamShaperXL Turbo v2 1.safetensors - Image Recognition MMproj: Pick the correct one for your model architecture here https://huggingface.co/koboldcpp/mmproj/tree/main - Speech Recognition: Whisper models for Speech-To-Text https://huggingface.co/koboldcpp/whisper/tree/main - Text-To-Speech: TTS models for Narration https://huggingface.co/koboldcpp/tts/tree/main - Text Generation: