{"slug": "bartowski-turns-gguf-quantization-heuristics-into-per-tensor-maps", "title": "Bartowski turns GGUF quantization heuristics into per-tensor maps", "summary": "Red Hat senior machine learning engineer Bartowski published a per-tensor layout map method for GGUF quantization on September 10th, replacing fixed name-, shape- and position-based precision rules with maps derived from measured sensitivity. Bartowski ran roughly 96 hours of experiments on a Framework Desktop with Claude Code, producing and evaluating more than 1,000 quantizations, and reported that token embeddings dominated sensitivity, causing about eight times the damage of the worst individual weight tensor in the 0.8B Qwen3.5 model and 16 times the damage in the 4B model. The results, measured by KL divergence against bf16 references using llama-perplexity on wikitext-2-raw at a 512 context length, have not been independently reproduced.", "body_md": "# Bartowski turns GGUF quantization heuristics into per-tensor maps\n\n**The Red Hat engineer says 1,000-plus tests found embeddings and several attention tensors deserve a larger share of the bit budget.**\n\n        By [RuntimeWire Staff](/author/runtimewire-staff)\n        · Published \n\nPrimary source: [Hugging Face Newsroom](https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization)\n\n## Why it matters\n\nGGUF files make open models practical on local hardware. Better bit allocation could preserve more model behavior at the same memory limit, though broad independent benchmarks are still needed.\n\n[Bartowski](https://huggingface.co/bartowski?ref=runtimewire) published a tensor-by-tensor method for assigning precision inside GGUF model files on September 10th, replacing years-old general rules in his quantization pipeline with maps derived from measured sensitivity.\n\nThe senior machine learning engineer at Red Hat is a familiar operator in local AI. His [Hugging Face profile](https://huggingface.co/bartowski?ref=runtimewire) is packed with compressed versions of open-weight models intended to run on consumer and workstation hardware. Quantization is the unglamorous engineering behind those releases: reduce the number of bits used to store model weights, fit the model into less memory, and try to preserve its behavior.\n\nBartowski's [new approach](https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization?ref=runtimewire) changes how that limited bit budget is distributed. Rather than deciding precision mainly from tensor names, shapes and positions using fixed rules, his pipeline can generate a layout map that specifies a quantization type for every eligible tensor.\n\nThe distinction matters because models have changed faster than the old rules beneath many GGUF releases. Bartowski traced the project to the spread of mixture-of-experts models, particularly architectures with more than eight experts and small `shexp` tensors that proved unusually sensitive. He had already maintained a modified version of `llama-quant.cpp` to handle those cases. New model families and competing layout methods convinced him that another collection of special cases would eventually become the same maintenance problem.\n\n### Four days of deliberately damaging models\n\nBartowski worked with Claude Code to build testing scripts, then said he ran roughly 96 hours of experiments on a [Framework Desktop](https://frame.work/desktop?ref=runtimewire), producing and evaluating more than 1,000 quantizations.\n\nThe central experiment was simple and computationally expensive. In a \"degrade-one\" run, Bartowski stored nearly every tensor at the relatively high-precision `q8_0` level while crushing one tensor to `q2_k`. The resulting change showed how much damage that tensor could cause. He also tried the inverse, upgrading one tensor inside an otherwise low-precision model, though he said those results contained too much noise to drive the method.\n\nThe main sweeps used Qwen3.5 models at 0.8B and 4B parameters. Bartowski then checked the patterns against models from other families, including Gemma 4, IBM's Granite 4.2, MiniCPM5, Ling, Muse and DeepSeek-V2-Lite.\n\nHis measurements compared the token probabilities produced by each quantized model with those of its bf16 reference using KL divergence, or KLD. The tests used `llama-perplexity` on wikitext-2-raw at a context length of 512. Lower KLD meant that the compressed model remained closer to the reference.\n\nThe results, which have not been independently reproduced, gave Bartowski a hierarchy of weak spots. He reported that token embeddings dominated the sensitivity scale, producing about eight times the damage of the worst individual weight tensor in the 0.8B model and 16 times the damage in the 4B model. Early and late layers followed the familiar U-shaped sensitivity curve. Attention value and output projections, feed-forward up projections and state-space-model output tensors also ranked highly. Feed-forward gate tensors generally did not earn additional bits in his tests.\n\nThose findings do not mean every embedding should remain at `q8_0`. Bartowski found that `q4_k` restored much of the lost performance, leaving the solver room to spend bits elsewhere when targeting smaller files.\n\n### A solver decides where each bit goes\n\nBartowski has published the generator, solver and frozen sensitivity data in his [quantization configuration repository](https://github.com/bartowski1182/quantization-config?ref=runtimewire). The solver takes a model's shape, measured sensitivity priors, GGML block-size information and a target quantization type. It produces a `--tensor-type-file` for [llama.cpp](https://github.com/ggml-org/llama.cpp?ref=runtimewire), assigning a type to each quantizable tensor rather than accepting the runtime's built-in heuristic.\n\nThe repository contains the public configuration, generator and sensitivity data behind the method. Bartowski's article also publishes the solver and layout-generation files used to produce the maps.\n\nBartowski also imposed clearer rules on GGUF's increasingly elastic suffixes. Under the new scheme, `_S` means at least 90% of a model's body tensors remain at the named quantization type. `_M` sets that floor at 70%, while `_L` permits up to half of those tensors to receive higher precision. Older variants including `Q2_K_L`, `Q3_K_XL` and `Q5_K_L` are being retired.\n\nThat cleanup exposes a practical problem with the previous naming system. Two files carrying different labels could converge toward similar bits-per-weight totals, while a supposedly lower-tier file could become larger than a higher-tier option after its embeddings were preserved at greater precision. Bartowski said the new layouts made `Q3_K_M` about 15% smaller on Qwen3.5-4B. In another test, `Q4_K_M` had slightly worse overall KLD while using just over 5% less space, giving it a better result per bit by his measure.\n\nThe goal is a menu where the label once again describes what is inside the file. Users choosing among quantizations can compare meaningful tiers instead of relying on suffixes accumulated through years of exceptions.\n\n### Release metadata makes the layouts inspectable\n\nThe method has already been used in releases for [MiniCPM5-2B](https://huggingface.co/bartowski/MiniCPM5-2B-GGUF?ref=runtimewire) and [Gryphe Pantheon Reasoning 26B](https://huggingface.co/bartowski/Gryphe_Pantheon-Reasoning-26B-A4B-1.1-V2-GGUF?ref=runtimewire), according to the available model-release records. Those releases show the layout method moving beyond the original test models.\n\nThe evidence remains bounded. Bartowski fit the original prior largely on two small Qwen models and evaluated KLD on one dataset. Independent tests would need to establish whether the allocation method improves task accuracy and perplexity across model families without harming inference speed or hardware compatibility.\n\nOther community projects are working on the same allocation problem. [Thireus's GGUF Tool Suite](https://github.com/Thireus/GGUF-Tool-Suite?ref=runtimewire) also measures the effect of degrading individual tensors and builds model-specific recipes. The field lacks a broad, independent comparison covering task accuracy, perplexity, inference speed and compatibility across hardware backends.\n\nBartowski's contribution is still concrete: a working pipeline, published sensitivity tables and reproducible release metadata. His next planned test is [Qwen3.8-27B](/models/qwen/qwen3.8-27b), followed by work on models with large n-gram tables and unusual multi-head latent attention designs.\n\nGGUF users will encounter the change through smaller files, altered tier names and quantizations that spend precision differently. Bartowski spent four days breaking tensors one at a time so the pipeline could stop pretending they all fail in the same way.", "url": "https://wpnews.pro/news/bartowski-turns-gguf-quantization-heuristics-into-per-tensor-maps", "canonical_source": "https://runtimewire.com/article/bartowski-per-tensor-layout-maps-gguf-quantization", "published_at": "2026-09-11 05:50:36+00:00", "updated_at": "2026-09-11 05:56:08.283081+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "ai-tools", "developer-tools"], "entities": ["Bartowski", "Red Hat", "Hugging Face", "Claude Code", "Framework Desktop", "Qwen3.5", "Gemma 4", "IBM Granite 4.2"], "alternates": {"html": "https://wpnews.pro/news/bartowski-turns-gguf-quantization-heuristics-into-per-tensor-maps", "markdown": "https://wpnews.pro/news/bartowski-turns-gguf-quantization-heuristics-into-per-tensor-maps.md", "text": "https://wpnews.pro/news/bartowski-turns-gguf-quantization-heuristics-into-per-tensor-maps.txt", "jsonld": "https://wpnews.pro/news/bartowski-turns-gguf-quantization-heuristics-into-per-tensor-maps.jsonld"}}