A new arXiv study (2608.15693v1) finds that no single model compression technique wins across edge AI tasks, reporting that Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization…
AIsbom v1.3.0, a tool for scanning ML model artifacts, now flags Keras Lambda layers and embedded code objects as CRITICAL, without unmarshalling payloads. The tool also detects ONNX external-data pat…
Unsloth Desktop, a new local AI training tool from Unsloth, enables users to train models locally without a cloud backend, supporting NVIDIA, AMD, Intel, and Mac hardware. It claims a 70% reduction in…
LiquidAI and collaborators released fuse-1 Lite, a 5.72 billion parameter mixture-of-experts model that transplants 960 coding experts from Qwen3.6-35B-A3B into LiquidAI's LFM2.5-2.6B host model, achi…
Fuse-1 Lite, a 5.72B parameter mixture-of-experts coding model from LiquidAI, can run locally with VRAM needs ranging from 3.36 GB in 4-bit quantized form to about 12 GB in full bfloat16 precision, ac…
An engineer details the GPU memory requirements for self-hosting large language models, explaining that the KV cache, not model weights, is the primary bottleneck under real traffic. The post provides…
Developer Antonello F. released Ferrox, a pure-Rust inference engine that runs open LLMs locally on CPU, Apple Metal, or CUDA, achieving performance parity with llama.cpp on an Apple M2 Pro: 26.9 tok/…
Empero released Qwythos-27B-v1, an open-weights reasoning model under Apache-2.0, built on Qwen3.5-27B with native multi-token prediction, full vision, and a 1,048,576-token context via YaRN. The 27B …
PrismML released the 1-bit Bonsai-27B language model, deployable via a specialized fork of llama.cpp with CUDA kernels for the Q1_0_g128 GGUF format. The model requires only ~5.2 GB peak memory at 4K …
A developer deploying the Kimi K3 model encountered a CUDA out-of-memory error caused by KV cache allocation during initial inference passes, not the model weights themselves. The developer resolved t…
A senior engineer with 11 years of distributed systems experience explains the full LLM inference pipeline, from request arrival to text output, detailing the GGUF file structure and the distinction b…
Open-weight AI models offer freedom but require significant effort to deploy, according to a technical guide that argues the real value lies in deployment pipelines and developer ecosystems rather tha…
Vendor lock-in in AI infrastructure creates technical debt and forces companies into a one-size-fits-all approach, warns a technical guide. To avoid this, the guide recommends implementing a Gateway P…
Kdeps, a CLI coding agent that runs on llamafile and/or GGUF, enables building and deploying AI agents in YAML with workflow (DAG pipelines) and agent (autonomous LLM loop) modes. The open-source tool…
Retort, a framework for comparing coding stacks, reveals that a model's performance depends on the entire stack—language, weights format, serving engine, agent, context engine, sampling, and prompt—no…
A developer deploying a fine-tuned Llama 3.2 3B medical AI model offline on Android for Yoruba, Hausa, Igbo, and Nigerian Pidgin encountered three distinct crashes during GGUF conversion—tokenizer mis…
A fork of the DwarfStar inference engine, andreaborio/ds4, aims to run large Mixture-of-Experts models like Qwen3.6-35B-A3B on 16–64 GB Apple Silicon Macs by using adaptive SSD streaming and Metal res…
Cogni-ML, a Crystal machine learning library, now supports native Apple Silicon GPU acceleration with Metal inference for GGUF models, including nomic-embed-text-v2-moe embeddings and Qwen 3.5 9B text…
A developer built GGUFun, a project that creates hand-crafted GGUF model files to run deterministic games like Snake and a first-person roaming demo on Ollama, with no training involved. The models us…
A new open-source tool, jlens-gguf, implements Anthropic's Jacobian Lens for GGUF models, enabling interactive visualization and live steering of language model activations directly through a web UI. …