Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way A developer quantized Google's MedGemma-1.5-4B medical vision-language model to INT4 (W4A16) using llm-compressor's GPTQModifier, reducing size from 8.6 GB to 5.2 GB for self-hosted deployment. The process required workarounds for a NotImplementedError in the official calibration example and careful dependency management to avoid torch conflicts. Quantized Google's MedGemma-1.5-4B a medical vision-language model to INT4 W4A16 via llm-compressor 's GPTQModifier, for self-hosted deployment. 8.6 GB in BF16 - 5.2 GB quantized. Full step-by-step below, model link at the bottom. References: gemma3 example.py model class, GPTQ/W4A16 recipe and gemma4 example.py calibration dataset pattern . GPTQ, via llm-compressor. Not AWQ. AutoAWQ is llm-compressor has the same gap for AWQ specifically here — confirmed via two open GitHub issues describing outright failures.Ran on RunPod RTX A5000 , using the instance's own pre-configured JupyterLab directly. nvidia-smi check actual CUDA version first pip3 uninstall torch torchvision -y pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128 python python -c "import torch; print torch. version ; print torch.cuda.is available " Confirm True before continuing. Then install llmcompressor separately, with --no-deps otherwise it silently pulls in a conflicting torch : pip3 install llmcompressor==0.12.0 --no-deps Then the other packages: pip3 install transformers==5.10.1 compressed-tensors==0.17.1 requests==2.32.5 \ pillow==11.0.0 zstandard==0.25.0 python-dotenv==1.2.2 Real final versions used: torch==2.11.0+cu128 , torchvision==0.26.0+cu128 , llmcompressor==0.12.0 , compressed-tensors==0.17.1 , transformers==5.10.1 , requests==2.32.5 , pillow==11.0.0 , zstandard==0.25.0 . Set up your HF token before you need it — MedGemma is gated. Create a .env : HF TOKEN=hf xxx python from dotenv import load dotenv load dotenv hf download google/medgemma-1.5-4b-it --local-dir ./medgemma-1.5-4b-it python from transformers import AutoProcessor, Gemma3ForConditionalGeneration MODEL ID = "./medgemma-1.5-4b-it" model = Gemma3ForConditionalGeneration.from pretrained MODEL ID processor = AutoProcessor.from pretrained MODEL ID We used dataset="flickr30k", splits={...} but hit: NotImplementedError: Label masking for vision datasets has not been implemented yet This is gemma3 example.py 's own approach — anyone following that example verbatim hits the same error. Fix: pretokenize calibration data manually instead the gemma4 example.py pattern : python from datasets import load dataset NUM CALIBRATION SAMPLES = 32 MAX SEQUENCE LENGTH = 2048 BATCH SIZE = 1 def get calib dataset processor : ds = load dataset "mit-han-lab/pile-val-backup", split=f"validation :{NUM CALIBRATION SAMPLES 10} " def preprocess example : return {"input ids": processor.tokenizer.encode example "text" .strip :MAX SEQUENCE LENGTH } return ds.shuffle seed=42 .map preprocess, remove columns=ds.column names .filter lambda ex: len ex "input ids" = MAX SEQUENCE LENGTH .select range NUM CALIBRATION SAMPLES Text-only, even for a vision model — the vision tower is excluded from quantization anyway, so calibration only needs to exercise the language-model layers. python from llmcompressor.modifiers.gptq import GPTQModifier recipe = GPTQModifier targets="Linear", scheme="W4A16", ignore= "lm head", r"re:model\.vision tower. ", r"re:model\.multi modal projector. " , , python from llmcompressor import oneshot oneshot model=model, processor=processor, dataset=get calib dataset processor , recipe=recipe, batch size=BATCH SIZE, shuffle calibration samples=False, max seq length=MAX SEQUENCE LENGTH, num calibration samples=NUM CALIBRATION SAMPLES, python from PIL import Image import requests messages = {"role": "user", "content": {"type": "text", "text": "Please describe what you see in this image\n"}, {"type": "image"}, }, prompt = processor.apply chat template messages, add generation prompt=True image url = "http://images.cocodataset.org/train2017/000000231895.jpg" raw image = Image.open requests.get image url, stream=True .raw print raw image.size, raw image.mode inputs = processor images=raw image, text=prompt, return tensors="pt" .to model.device print inputs.keys Confirm pixel values is in there. Real gotcha: it's easy to write processor image=raw image, ... — singular — instead of images= plural . Get this wrong and the image silently never reaches the model, but generation doesn't error — it just confidently describes something completely unrelated. Caught us for a bit. Check this before trusting any output. python from compressed tensors.offload import dispatch model dispatch model model compile disabled — known issue: huggingface/transformers 38333 output = model.generate inputs, max new tokens=1024, disable compile=True print processor.decode output 0 , skip special tokens=True Confirm the description actually matches the image. Also re-ran this against a real chest X-ray — MedGemma's vision encoder is trained only on medical imagery, so a generic photo confirms plumbing but isn't a fair capability test. QUANTIZED MODEL ID = MODEL ID.rstrip "/" .split "/" -1 + "-W4A16-G128" model.save pretrained QUANTIZED MODEL ID, save compressed=True do NOT call processor.save pretrained here — see Step 10 Calling processor.save pretrained on this model renames extra special tokens to a non-standard model specific special tokens key and drops added tokens decoder entirely — a documented transformers round-trip bug also seen on Phi-4-mini . Copy the files directly from the original instead: python import shutil from pathlib import Path model path = Path "./medgemma-1.5-4b-it" quantized model path = Path f"./{QUANTIZED MODEL ID}" files to copy = "tokenizer.json", "tokenizer config.json", "special tokens map.json", "preprocessor config.json", "chat template.json", for fname in files to copy: src = model path / fname if src.exists : shutil.copy src, quantized model path / fname Gemma3's newer nested rope parameters format full attention / sliding attention sub-dicts isn't recognized by some downstream tooling expecting a flat schema with a top-level rope type key. The key existed before, just nested — this restructures it, it doesn't add something absent. If you're following along in a notebook, add %%writefile patch rope config.py as the first line of the cell to write it straight to disk: python import json import shutil import sys from pathlib import Path def patch config config path: str, dominant rope type: str = "default" : config path = Path config path backup path = config path.with suffix ".json.bak" if not backup path.exists : shutil.copy2 config path, backup path print f"Backed up original to {backup path}" else: print f"Backup already exists at {backup path}, not overwriting it" with open config path, "r" as f: config = json.load f text config = config.get "text config" if text config is None: print "ERROR: no 'text config' block found — is this the right file?" sys.exit 1 rope params = text config.get "rope parameters" if rope params is None: print "ERROR: no 'rope parameters' found under text config" sys.exit 1 if "rope type" in rope params: print "'rope type' key already present at top level — nothing to do" return new rope params = {"rope type": dominant rope type} new rope params.update rope params text config "rope parameters" = new rope params with open config path, "w" as f: json.dump config, f, indent=2 print f"Patched {config path}: added top-level rope type='{dominant rope type}' " f"inside rope parameters full attention/sliding attention left untouched " if name == " main ": if len sys.argv < 2: print "Usage: python patch rope config.py /path/to/config.json rope type " sys.exit 1 path = sys.argv 1 rtype = sys.argv 2 if len sys.argv 2 else "default" patch config path, rtype python patch rope config.py ./medgemma-1.5-4b-it-W4A16-G128/config.json Keeps a .json.bak backup before touching anything, and is idempotent — safe to re-run if you're not sure whether it's already been applied. A known llm-compressor bug issue 1546 https://github.com/vllm-project/llm-compressor/issues/1546 : transformers =4.52's multimodal refactor changed internal weight naming during model loading, but llm-compressor 's quantization config.ignore list is generated from the converted naming scheme without being reconciled against the actual saved safetensors key names. Left uncorrected, the ignore list matches no real tensor — the vision tower would silently get quantized despite the recipe saying otherwise. python import json def fix model config config path : with open config path as f: config = json.load f ignore list = config "quantization config" "ignore" new ignore = for entry in ignore list: if entry.startswith "model.vision tower." : fixed = entry.replace "model.vision tower.", "vision tower.vision model.", 1 new ignore.append fixed else: new ignore.append entry e.g. "lm head" stays as-is config "quantization config" "ignore" = new ignore with open config path, "w" as f: json.dump config, f, indent=2 print "Sample before:", ignore list 0 print "Sample after: ", new ignore 0 fix model config "medgemma-1.5-4b-it-W4A16-G128/config.json" If you set HF TOKEN in your .env back in Step 2, huggingface hub picks it up automatically. Otherwise, hf auth login first. Don't use model.push to hub — it calls save pretrained again KeyError: 'weight' no plain weight key left to compress a second python from huggingface hub import HfApi REPO ID = f"