# Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way

> Source: <https://dev.to/joteqthefirst/quantizing-medgemma-to-int4-gptqw4a16-everything-that-broke-along-the-way-2a9f>
> Published: 2026-07-14 10:48:23+00:00

Quantized Google's MedGemma-1.5-4B (a medical vision-language

model) to INT4 (W4A16) via `llm-compressor`

's GPTQModifier, for

self-hosted deployment. 8.6 GB in BF16 -> 5.2 GB quantized. Full

step-by-step below, model link at the bottom.

**References:** `gemma3_example.py`

(model class, GPTQ/W4A16 recipe) and

`gemma4_example.py`

(calibration dataset pattern).

**GPTQ, via llm-compressor. Not AWQ.**

`AutoAWQ`

is `llm-compressor`

has the same gap for AWQ
specifically here — confirmed via two open GitHub issues describing
outright failures.Ran on RunPod (RTX A5000), using the instance's own pre-configured

JupyterLab directly.

```
nvidia-smi   # check actual CUDA version first
pip3 uninstall torch torchvision -y
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128
python
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
```

Confirm `True`

before continuing. Then install `llmcompressor`

separately, with `--no-deps`

(otherwise it silently pulls in a

conflicting `torch`

):

```
pip3 install llmcompressor==0.12.0 --no-deps
```

Then the other packages:

```
pip3 install transformers==5.10.1 compressed-tensors==0.17.1 requests==2.32.5 \
  pillow==11.0.0 zstandard==0.25.0 python-dotenv==1.2.2
```

Real final versions used: `torch==2.11.0+cu128`

,

`torchvision==0.26.0+cu128`

, `llmcompressor==0.12.0`

,

`compressed-tensors==0.17.1`

, `transformers==5.10.1`

,

`requests==2.32.5`

, `pillow==11.0.0`

, `zstandard==0.25.0`

.

Set up your HF token before you need it — MedGemma is gated. Create a

`.env`

:

```
HF_TOKEN=hf_xxx
python
from dotenv import load_dotenv
load_dotenv()
hf download google/medgemma-1.5-4b-it --local-dir ./medgemma-1.5-4b-it
python
from transformers import AutoProcessor, Gemma3ForConditionalGeneration

MODEL_ID = "./medgemma-1.5-4b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)
```

We used `dataset="flickr30k", splits={...}`

but hit:

```
NotImplementedError: Label masking for vision datasets has not been implemented yet
```

This is `gemma3_example.py`

's own approach — anyone following that

example verbatim hits the same error. Fix: pretokenize calibration data

manually instead (the `gemma4_example.py`

pattern):

``` python
from datasets import load_dataset

NUM_CALIBRATION_SAMPLES = 32
MAX_SEQUENCE_LENGTH = 2048
BATCH_SIZE = 1

def get_calib_dataset(processor):
    ds = load_dataset("mit-han-lab/pile-val-backup", split=f"validation[:{NUM_CALIBRATION_SAMPLES * 10}]")
    def preprocess(example):
        return {"input_ids": processor.tokenizer.encode(example["text"].strip())[:MAX_SEQUENCE_LENGTH]}
    return (
        ds.shuffle(seed=42)
        .map(preprocess, remove_columns=ds.column_names)
        .filter(lambda ex: len(ex["input_ids"]) >= MAX_SEQUENCE_LENGTH)
        .select(range(NUM_CALIBRATION_SAMPLES))
    )
```

Text-only, even for a vision model — the vision tower is excluded from

quantization anyway, so calibration only needs to exercise the

language-model layers.

``` python
from llmcompressor.modifiers.gptq import GPTQModifier

recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        ignore=[
            "lm_head",
            r"re:model\.vision_tower.*",
            r"re:model\.multi_modal_projector.*"
        ],
    ),
]
python
from llmcompressor import oneshot

oneshot(
    model=model,
    processor=processor,
    dataset=get_calib_dataset(processor),
    recipe=recipe,
    batch_size=BATCH_SIZE,
    shuffle_calibration_samples=False,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)
python
from PIL import Image
import requests

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": "Please describe what you see in this image\n"},
        {"type": "image"},
    ]},
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)

image_url = "http://images.cocodataset.org/train2017/000000231895.jpg"
raw_image = Image.open(requests.get(image_url, stream=True).raw)
print(raw_image.size, raw_image.mode)

inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to(model.device)
print(inputs.keys())
```

Confirm `pixel_values`

is in there. **Real gotcha:** it's easy to write

`processor(image=raw_image, ...)`

— singular — instead of `images=`

(plural). Get this wrong and the image silently never reaches the

model, but generation doesn't error — it just confidently describes

something completely unrelated. Caught us for a bit. Check this before

trusting any output.

``` python
from compressed_tensors.offload import dispatch_model

dispatch_model(model)

# compile disabled — known issue: huggingface/transformers#38333
output = model.generate(**inputs, max_new_tokens=1024, disable_compile=True)
print(processor.decode(output[0], skip_special_tokens=True))
```

Confirm the description actually matches the image. Also re-ran this

against a real chest X-ray — MedGemma's vision encoder is trained only

on medical imagery, so a generic photo confirms plumbing but isn't a

fair capability test.

```
QUANTIZED_MODEL_ID = MODEL_ID.rstrip("/").split("/")[-1] + "-W4A16-G128"
model.save_pretrained(QUANTIZED_MODEL_ID, save_compressed=True)
# do NOT call processor.save_pretrained() here — see Step 10
```

Calling `processor.save_pretrained()`

on this model renames

`extra_special_tokens`

to a non-standard `model_specific_special_tokens`

key and drops `added_tokens_decoder`

entirely — a documented

`transformers`

round-trip bug (also seen on Phi-4-mini). Copy the files

directly from the original instead:

``` python
import shutil
from pathlib import Path

model_path = Path("./medgemma-1.5-4b-it")
quantized_model_path = Path(f"./{QUANTIZED_MODEL_ID}")

files_to_copy = [
    "tokenizer.json",
    "tokenizer_config.json",
    "special_tokens_map.json",
    "preprocessor_config.json",
    "chat_template.json",
]

for fname in files_to_copy:
    src = model_path / fname
    if src.exists():
        shutil.copy(src, quantized_model_path / fname)
```

Gemma3's newer nested `rope_parameters`

format (`full_attention`

/

`sliding_attention`

sub-dicts) isn't recognized by some downstream

tooling expecting a flat schema with a top-level `rope_type`

key. The

key existed before, just nested — this restructures it, it doesn't add

something absent. If you're following along in a notebook, add

`%%writefile patch_rope_config.py`

as the first line of the cell to

write it straight to disk:

``` python
import json
import shutil
import sys
from pathlib import Path

def patch_config(config_path: str, dominant_rope_type: str = "default"):
    config_path = Path(config_path)
    backup_path = config_path.with_suffix(".json.bak")

    if not backup_path.exists():
        shutil.copy2(config_path, backup_path)
        print(f"Backed up original to {backup_path}")
    else:
        print(f"Backup already exists at {backup_path}, not overwriting it")

    with open(config_path, "r") as f:
        config = json.load(f)

    text_config = config.get("text_config")
    if text_config is None:
        print("ERROR: no 'text_config' block found — is this the right file?")
        sys.exit(1)

    rope_params = text_config.get("rope_parameters")
    if rope_params is None:
        print("ERROR: no 'rope_parameters' found under text_config")
        sys.exit(1)

    if "rope_type" in rope_params:
        print("'rope_type' key already present at top level — nothing to do")
        return

    new_rope_params = {"rope_type": dominant_rope_type}
    new_rope_params.update(rope_params)
    text_config["rope_parameters"] = new_rope_params

    with open(config_path, "w") as f:
        json.dump(config, f, indent=2)

    print(f"Patched {config_path}: added top-level rope_type='{dominant_rope_type}' "
          f"inside rope_parameters (full_attention/sliding_attention left untouched)")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python patch_rope_config.py /path/to/config.json [rope_type]")
        sys.exit(1)
    path = sys.argv[1]
    rtype = sys.argv[2] if len(sys.argv) > 2 else "default"
    patch_config(path, rtype)
python patch_rope_config.py ./medgemma-1.5-4b-it-W4A16-G128/config.json
```

Keeps a `.json.bak`

backup before touching anything, and is idempotent

— safe to re-run if you're not sure whether it's already been applied.

A known `llm-compressor`

bug ([issue #1546](https://github.com/vllm-project/llm-compressor/issues/1546)):

`transformers`

>=4.52's multimodal refactor changed internal weight

naming during model loading, but `llm-compressor`

's

`quantization_config.ignore`

list is generated from the converted naming

scheme without being reconciled against the actual saved safetensors key

names. Left uncorrected, the ignore list matches no real tensor — the

vision tower would silently get quantized despite the recipe saying

otherwise.

``` python
import json

def fix_model_config(config_path):
    with open(config_path) as f:
        config = json.load(f)

    ignore_list = config["quantization_config"]["ignore"]

    new_ignore = []
    for entry in ignore_list:
        if entry.startswith("model.vision_tower."):
            fixed = entry.replace("model.vision_tower.", "vision_tower.vision_model.", 1)
            new_ignore.append(fixed)
        else:
            new_ignore.append(entry)  # e.g. "lm_head" stays as-is

    config["quantization_config"]["ignore"] = new_ignore

    with open(config_path, "w") as f:
        json.dump(config, f, indent=2)

    print("Sample before:", ignore_list[0])
    print("Sample after: ", new_ignore[0])

fix_model_config("medgemma-1.5-4b-it-W4A16-G128/config.json")
```

If you set `HF_TOKEN`

in your `.env`

back in Step 2, `huggingface_hub`

picks it up automatically. Otherwise, `hf auth login`

first.

**Don't use model.push_to_hub()** — it calls

`save_pretrained()`

again`KeyError: 'weight'`

(no plain `weight`

key left to compress a second

``` python
from huggingface_hub import HfApi

REPO_ID = f"<your-hf-username-or-org>/{QUANTIZED_MODEL_ID}"
api = HfApi()
api.create_repo(REPO_ID, exist_ok=True)
api.upload_folder(folder_path=QUANTIZED_MODEL_ID, repo_id=REPO_ID)
```

**Result:** 8.6 GB in BF16 -> 5.2 GB quantized. Vision tower,

multimodal projector, and `lm_head`

stay unquantized, which is why it's

not the full theoretical 75% reduction.

Model: `confamnode/medgemma-1.5-4b-it-W4A16-G128`

More technical blogs at `ConfamNode`