{"slug": "quantizing-medgemma-to-int4-gptq-w4a16-everything-that-broke-along-the-way", "title": "Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way", "summary": "A developer quantized Google's MedGemma-1.5-4B medical vision-language model to INT4 (W4A16) using llm-compressor's GPTQModifier, reducing size from 8.6 GB to 5.2 GB for self-hosted deployment. The process required workarounds for a NotImplementedError in the official calibration example and careful dependency management to avoid torch conflicts.", "body_md": "Quantized Google's MedGemma-1.5-4B (a medical vision-language\n\nmodel) to INT4 (W4A16) via `llm-compressor`\n\n's GPTQModifier, for\n\nself-hosted deployment. 8.6 GB in BF16 -> 5.2 GB quantized. Full\n\nstep-by-step below, model link at the bottom.\n\n**References:** `gemma3_example.py`\n\n(model class, GPTQ/W4A16 recipe) and\n\n`gemma4_example.py`\n\n(calibration dataset pattern).\n\n**GPTQ, via llm-compressor. Not AWQ.**\n\n`AutoAWQ`\n\nis `llm-compressor`\n\nhas the same gap for AWQ\nspecifically here — confirmed via two open GitHub issues describing\noutright failures.Ran on RunPod (RTX A5000), using the instance's own pre-configured\n\nJupyterLab directly.\n\n```\nnvidia-smi   # check actual CUDA version first\npip3 uninstall torch torchvision -y\npip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128\npython\npython -c \"import torch; print(torch.__version__); print(torch.cuda.is_available())\"\n```\n\nConfirm `True`\n\nbefore continuing. Then install `llmcompressor`\n\nseparately, with `--no-deps`\n\n(otherwise it silently pulls in a\n\nconflicting `torch`\n\n):\n\n```\npip3 install llmcompressor==0.12.0 --no-deps\n```\n\nThen the other packages:\n\n```\npip3 install transformers==5.10.1 compressed-tensors==0.17.1 requests==2.32.5 \\\n  pillow==11.0.0 zstandard==0.25.0 python-dotenv==1.2.2\n```\n\nReal final versions used: `torch==2.11.0+cu128`\n\n,\n\n`torchvision==0.26.0+cu128`\n\n, `llmcompressor==0.12.0`\n\n,\n\n`compressed-tensors==0.17.1`\n\n, `transformers==5.10.1`\n\n,\n\n`requests==2.32.5`\n\n, `pillow==11.0.0`\n\n, `zstandard==0.25.0`\n\n.\n\nSet up your HF token before you need it — MedGemma is gated. Create a\n\n`.env`\n\n:\n\n```\nHF_TOKEN=hf_xxx\npython\nfrom dotenv import load_dotenv\nload_dotenv()\nhf download google/medgemma-1.5-4b-it --local-dir ./medgemma-1.5-4b-it\npython\nfrom transformers import AutoProcessor, Gemma3ForConditionalGeneration\n\nMODEL_ID = \"./medgemma-1.5-4b-it\"\nmodel = Gemma3ForConditionalGeneration.from_pretrained(MODEL_ID)\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\n```\n\nWe used `dataset=\"flickr30k\", splits={...}`\n\nbut hit:\n\n```\nNotImplementedError: Label masking for vision datasets has not been implemented yet\n```\n\nThis is `gemma3_example.py`\n\n's own approach — anyone following that\n\nexample verbatim hits the same error. Fix: pretokenize calibration data\n\nmanually instead (the `gemma4_example.py`\n\npattern):\n\n``` python\nfrom datasets import load_dataset\n\nNUM_CALIBRATION_SAMPLES = 32\nMAX_SEQUENCE_LENGTH = 2048\nBATCH_SIZE = 1\n\ndef get_calib_dataset(processor):\n    ds = load_dataset(\"mit-han-lab/pile-val-backup\", split=f\"validation[:{NUM_CALIBRATION_SAMPLES * 10}]\")\n    def preprocess(example):\n        return {\"input_ids\": processor.tokenizer.encode(example[\"text\"].strip())[:MAX_SEQUENCE_LENGTH]}\n    return (\n        ds.shuffle(seed=42)\n        .map(preprocess, remove_columns=ds.column_names)\n        .filter(lambda ex: len(ex[\"input_ids\"]) >= MAX_SEQUENCE_LENGTH)\n        .select(range(NUM_CALIBRATION_SAMPLES))\n    )\n```\n\nText-only, even for a vision model — the vision tower is excluded from\n\nquantization anyway, so calibration only needs to exercise the\n\nlanguage-model layers.\n\n``` python\nfrom llmcompressor.modifiers.gptq import GPTQModifier\n\nrecipe = [\n    GPTQModifier(\n        targets=\"Linear\",\n        scheme=\"W4A16\",\n        ignore=[\n            \"lm_head\",\n            r\"re:model\\.vision_tower.*\",\n            r\"re:model\\.multi_modal_projector.*\"\n        ],\n    ),\n]\npython\nfrom llmcompressor import oneshot\n\noneshot(\n    model=model,\n    processor=processor,\n    dataset=get_calib_dataset(processor),\n    recipe=recipe,\n    batch_size=BATCH_SIZE,\n    shuffle_calibration_samples=False,\n    max_seq_length=MAX_SEQUENCE_LENGTH,\n    num_calibration_samples=NUM_CALIBRATION_SAMPLES,\n)\npython\nfrom PIL import Image\nimport requests\n\nmessages = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"text\", \"text\": \"Please describe what you see in this image\\n\"},\n        {\"type\": \"image\"},\n    ]},\n]\nprompt = processor.apply_chat_template(messages, add_generation_prompt=True)\n\nimage_url = \"http://images.cocodataset.org/train2017/000000231895.jpg\"\nraw_image = Image.open(requests.get(image_url, stream=True).raw)\nprint(raw_image.size, raw_image.mode)\n\ninputs = processor(images=raw_image, text=prompt, return_tensors=\"pt\").to(model.device)\nprint(inputs.keys())\n```\n\nConfirm `pixel_values`\n\nis in there. **Real gotcha:** it's easy to write\n\n`processor(image=raw_image, ...)`\n\n— singular — instead of `images=`\n\n(plural). Get this wrong and the image silently never reaches the\n\nmodel, but generation doesn't error — it just confidently describes\n\nsomething completely unrelated. Caught us for a bit. Check this before\n\ntrusting any output.\n\n``` python\nfrom compressed_tensors.offload import dispatch_model\n\ndispatch_model(model)\n\n# compile disabled — known issue: huggingface/transformers#38333\noutput = model.generate(**inputs, max_new_tokens=1024, disable_compile=True)\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\nConfirm the description actually matches the image. Also re-ran this\n\nagainst a real chest X-ray — MedGemma's vision encoder is trained only\n\non medical imagery, so a generic photo confirms plumbing but isn't a\n\nfair capability test.\n\n```\nQUANTIZED_MODEL_ID = MODEL_ID.rstrip(\"/\").split(\"/\")[-1] + \"-W4A16-G128\"\nmodel.save_pretrained(QUANTIZED_MODEL_ID, save_compressed=True)\n# do NOT call processor.save_pretrained() here — see Step 10\n```\n\nCalling `processor.save_pretrained()`\n\non this model renames\n\n`extra_special_tokens`\n\nto a non-standard `model_specific_special_tokens`\n\nkey and drops `added_tokens_decoder`\n\nentirely — a documented\n\n`transformers`\n\nround-trip bug (also seen on Phi-4-mini). Copy the files\n\ndirectly from the original instead:\n\n``` python\nimport shutil\nfrom pathlib import Path\n\nmodel_path = Path(\"./medgemma-1.5-4b-it\")\nquantized_model_path = Path(f\"./{QUANTIZED_MODEL_ID}\")\n\nfiles_to_copy = [\n    \"tokenizer.json\",\n    \"tokenizer_config.json\",\n    \"special_tokens_map.json\",\n    \"preprocessor_config.json\",\n    \"chat_template.json\",\n]\n\nfor fname in files_to_copy:\n    src = model_path / fname\n    if src.exists():\n        shutil.copy(src, quantized_model_path / fname)\n```\n\nGemma3's newer nested `rope_parameters`\n\nformat (`full_attention`\n\n/\n\n`sliding_attention`\n\nsub-dicts) isn't recognized by some downstream\n\ntooling expecting a flat schema with a top-level `rope_type`\n\nkey. The\n\nkey existed before, just nested — this restructures it, it doesn't add\n\nsomething absent. If you're following along in a notebook, add\n\n`%%writefile patch_rope_config.py`\n\nas the first line of the cell to\n\nwrite it straight to disk:\n\n``` python\nimport json\nimport shutil\nimport sys\nfrom pathlib import Path\n\ndef patch_config(config_path: str, dominant_rope_type: str = \"default\"):\n    config_path = Path(config_path)\n    backup_path = config_path.with_suffix(\".json.bak\")\n\n    if not backup_path.exists():\n        shutil.copy2(config_path, backup_path)\n        print(f\"Backed up original to {backup_path}\")\n    else:\n        print(f\"Backup already exists at {backup_path}, not overwriting it\")\n\n    with open(config_path, \"r\") as f:\n        config = json.load(f)\n\n    text_config = config.get(\"text_config\")\n    if text_config is None:\n        print(\"ERROR: no 'text_config' block found — is this the right file?\")\n        sys.exit(1)\n\n    rope_params = text_config.get(\"rope_parameters\")\n    if rope_params is None:\n        print(\"ERROR: no 'rope_parameters' found under text_config\")\n        sys.exit(1)\n\n    if \"rope_type\" in rope_params:\n        print(\"'rope_type' key already present at top level — nothing to do\")\n        return\n\n    new_rope_params = {\"rope_type\": dominant_rope_type}\n    new_rope_params.update(rope_params)\n    text_config[\"rope_parameters\"] = new_rope_params\n\n    with open(config_path, \"w\") as f:\n        json.dump(config, f, indent=2)\n\n    print(f\"Patched {config_path}: added top-level rope_type='{dominant_rope_type}' \"\n          f\"inside rope_parameters (full_attention/sliding_attention left untouched)\")\n\nif __name__ == \"__main__\":\n    if len(sys.argv) < 2:\n        print(\"Usage: python patch_rope_config.py /path/to/config.json [rope_type]\")\n        sys.exit(1)\n    path = sys.argv[1]\n    rtype = sys.argv[2] if len(sys.argv) > 2 else \"default\"\n    patch_config(path, rtype)\npython patch_rope_config.py ./medgemma-1.5-4b-it-W4A16-G128/config.json\n```\n\nKeeps a `.json.bak`\n\nbackup before touching anything, and is idempotent\n\n— safe to re-run if you're not sure whether it's already been applied.\n\nA known `llm-compressor`\n\nbug ([issue #1546](https://github.com/vllm-project/llm-compressor/issues/1546)):\n\n`transformers`\n\n>=4.52's multimodal refactor changed internal weight\n\nnaming during model loading, but `llm-compressor`\n\n's\n\n`quantization_config.ignore`\n\nlist is generated from the converted naming\n\nscheme without being reconciled against the actual saved safetensors key\n\nnames. Left uncorrected, the ignore list matches no real tensor — the\n\nvision tower would silently get quantized despite the recipe saying\n\notherwise.\n\n``` python\nimport json\n\ndef fix_model_config(config_path):\n    with open(config_path) as f:\n        config = json.load(f)\n\n    ignore_list = config[\"quantization_config\"][\"ignore\"]\n\n    new_ignore = []\n    for entry in ignore_list:\n        if entry.startswith(\"model.vision_tower.\"):\n            fixed = entry.replace(\"model.vision_tower.\", \"vision_tower.vision_model.\", 1)\n            new_ignore.append(fixed)\n        else:\n            new_ignore.append(entry)  # e.g. \"lm_head\" stays as-is\n\n    config[\"quantization_config\"][\"ignore\"] = new_ignore\n\n    with open(config_path, \"w\") as f:\n        json.dump(config, f, indent=2)\n\n    print(\"Sample before:\", ignore_list[0])\n    print(\"Sample after: \", new_ignore[0])\n\nfix_model_config(\"medgemma-1.5-4b-it-W4A16-G128/config.json\")\n```\n\nIf you set `HF_TOKEN`\n\nin your `.env`\n\nback in Step 2, `huggingface_hub`\n\npicks it up automatically. Otherwise, `hf auth login`\n\nfirst.\n\n**Don't use model.push_to_hub()** — it calls\n\n`save_pretrained()`\n\nagain`KeyError: 'weight'`\n\n(no plain `weight`\n\nkey left to compress a second\n\n``` python\nfrom huggingface_hub import HfApi\n\nREPO_ID = f\"<your-hf-username-or-org>/{QUANTIZED_MODEL_ID}\"\napi = HfApi()\napi.create_repo(REPO_ID, exist_ok=True)\napi.upload_folder(folder_path=QUANTIZED_MODEL_ID, repo_id=REPO_ID)\n```\n\n**Result:** 8.6 GB in BF16 -> 5.2 GB quantized. Vision tower,\n\nmultimodal projector, and `lm_head`\n\nstay unquantized, which is why it's\n\nnot the full theoretical 75% reduction.\n\nModel: `confamnode/medgemma-1.5-4b-it-W4A16-G128`\n\nMore technical blogs at `ConfamNode`", "url": "https://wpnews.pro/news/quantizing-medgemma-to-int4-gptq-w4a16-everything-that-broke-along-the-way", "canonical_source": "https://dev.to/joteqthefirst/quantizing-medgemma-to-int4-gptqw4a16-everything-that-broke-along-the-way-2a9f", "published_at": "2026-07-14 10:48:23+00:00", "updated_at": "2026-07-14 10:58:03.555314+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Google", "MedGemma-1.5-4B", "llm-compressor", "GPTQModifier", "RunPod", "RTX A5000", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/quantizing-medgemma-to-int4-gptq-w4a16-everything-that-broke-along-the-way", "markdown": "https://wpnews.pro/news/quantizing-medgemma-to-int4-gptq-w4a16-everything-that-broke-along-the-way.md", "text": "https://wpnews.pro/news/quantizing-medgemma-to-int4-gptq-w4a16-everything-that-broke-along-the-way.txt", "jsonld": "https://wpnews.pro/news/quantizing-medgemma-to-int4-gptq-w4a16-everything-that-broke-along-the-way.jsonld"}}