A developer outlines how to build a lightweight, native coding agent in under 100 lines of code, using small open-source models like CodeGemma 2B or TinyLlama 1.1B and quantization for efficiency. The…
The GGML organization released Llama-macOS, a 4 MB macOS menu bar app that runs local LLMs via llama.cpp, featuring a local server at http://localhost:9931/v1, automatic model installation, and one-cl…
A developer comparing C++ and Rust for AI/ML code with LLMs found that while Rust's Burn framework is competitive on GPU, it performs poorly on CPU compared to C++ with GGML. The experiments, publishe…
Security researchers have found that using code diffs as a slicing mechanism significantly improves LLMs' ability to find vulnerabilities, according to a team exploring epistemological heuristics for …
A developer shared a configuration example for running Qwen 3.6 with llama-turboquant, a Docker-based CUDA inference container. The setup includes GPU offloading, flash attention, speculative decoding…
A developer provides a technical guide for compiling llama.cpp on Debian 12/13, covering native builds and cross-compilation from x86_64 to ARM64. The guide includes steps for enabling OpenBLAS and ve…
A developer discovered that the Tesla P100 GPU has been performing inaccurate floating-point arithmetic in llama.cpp for years due to a missing architecture check. A three-line patch that excludes the…
KoboldCpp, a single-file executable for running GGUF and GGML AI models, has been released by developer LostRuins. The tool requires no installation and supports text generation, image generation, spe…
A developer successfully quantized Google's Gemma 4 model to 4-bit precision using llama.cpp on a Mac with Metal acceleration. The process involved converting the model to GGUF format and applying the…
Ollama released version 0.30.0-rc23, a pre-release that shifts the software's architecture to directly support llama.cpp instead of building on top of GGML, and adds compatibility with the GGUF file f…
The article provides a technical guide for running AI models like Flux Schnell (12B) and LLMs on a legacy AMD RX 580 (8GB) GPU using native Vulkan support, bypassing deprecated CUDA and ROCm ecosystem…