Quantized Google's MedGemma-1.5-4B (a medical vision-language
model) to INT4 (W4A16) via llm-compressor
's GPTQModifier, for
self-hosted deployment. 8.6 GB in BF16 -> 5.2 GB quantized. Full
step-by-step below, model link at the bottom.
References: gemma3_example.py
(model class, GPTQ/W4A16 recipe) and
gemma4_example.py
(calibration dataset pattern).
GPTQ, via llm-compressor. Not AWQ.
AutoAWQ
is llm-compressor
has the same gap for AWQ specifically here ā confirmed via two open GitHub issues describing outright failures.Ran on RunPod (RTX A5000), using the instance's own pre-configured
JupyterLab directly.
nvidia-smi # check actual CUDA version first
pip3 uninstall torch torchvision -y
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128
python
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
Confirm True
before continuing. Then install llmcompressor
separately, with --no-deps
(otherwise it silently pulls in a
conflicting torch
):
pip3 install llmcompressor==0.12.0 --no-deps
Then the other packages:
pip3 install transformers==5.10.1 compressed-tensors==0.17.1 requests==2.32.5 \
pillow==11.0.0 zstandard==0.25.0 python-dotenv==1.2.2
Real final versions used: torch==2.11.0+cu128
,
torchvision==0.26.0+cu128
, llmcompressor==0.12.0
,
compressed-tensors==0.17.1
, transformers==5.10.1
,
requests==2.32.5
, pillow==11.0.0
, zstandard==0.25.0
.
Set up your HF token before you need it ā MedGemma is gated. Create a
.env
:
HF_TOKEN=hf_xxx
python
from dotenv import load_dotenv
load_dotenv()
hf download google/medgemma-1.5-4b-it --local-dir ./medgemma-1.5-4b-it
python
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
MODEL_ID = "./medgemma-1.5-4b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)
We used dataset="flickr30k", splits={...}
but hit:
NotImplementedError: Label masking for vision datasets has not been implemented yet
This is gemma3_example.py
's own approach ā anyone following that
example verbatim hits the same error. Fix: pretokenize calibration data
manually instead (the gemma4_example.py
pattern):
from datasets import load_dataset
NUM_CALIBRATION_SAMPLES = 32
MAX_SEQUENCE_LENGTH = 2048
BATCH_SIZE = 1
def get_calib_dataset(processor):
ds = load_dataset("mit-han-lab/pile-val-backup", split=f"validation[:{NUM_CALIBRATION_SAMPLES * 10}]")
def preprocess(example):
return {"input_ids": processor.tokenizer.encode(example["text"].strip())[:MAX_SEQUENCE_LENGTH]}
return (
ds.shuffle(seed=42)
.map(preprocess, remove_columns=ds.column_names)
.filter(lambda ex: len(ex["input_ids"]) >= MAX_SEQUENCE_LENGTH)
.select(range(NUM_CALIBRATION_SAMPLES))
)
Text-only, even for a vision model ā the vision tower is excluded from
quantization anyway, so calibration only needs to exercise the
language-model layers.
from llmcompressor.modifiers.gptq import GPTQModifier
recipe = [
GPTQModifier(
targets="Linear",
scheme="W4A16",
ignore=[
"lm_head",
r"re:model\.vision_tower.*",
r"re:model\.multi_modal_projector.*"
],
),
]
python
from llmcompressor import oneshot
oneshot(
model=model,
processor=processor,
dataset=get_calib_dataset(processor),
recipe=recipe,
batch_size=BATCH_SIZE,
shuffle_calibration_samples=False,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)
python
from PIL import Image
import requests
messages = [
{"role": "user", "content": [
{"type": "text", "text": "Please describe what you see in this image\n"},
{"type": "image"},
]},
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
image_url = "http://images.cocodataset.org/train2017/000000231895.jpg"
raw_image = Image.open(requests.get(image_url, stream=True).raw)
print(raw_image.size, raw_image.mode)
inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to(model.device)
print(inputs.keys())
Confirm pixel_values
is in there. Real gotcha: it's easy to write
processor(image=raw_image, ...)
ā singular ā instead of images=
(plural). Get this wrong and the image silently never reaches the
model, but generation doesn't error ā it just confidently describes
something completely unrelated. Caught us for a bit. Check this before
trusting any output.
from compressed_tensors.offload import dispatch_model
dispatch_model(model)
output = model.generate(**inputs, max_new_tokens=1024, disable_compile=True)
print(processor.decode(output[0], skip_special_tokens=True))
Confirm the description actually matches the image. Also re-ran this
against a real chest X-ray ā MedGemma's vision encoder is trained only
on medical imagery, so a generic photo confirms plumbing but isn't a
fair capability test.
QUANTIZED_MODEL_ID = MODEL_ID.rstrip("/").split("/")[-1] + "-W4A16-G128"
model.save_pretrained(QUANTIZED_MODEL_ID, save_compressed=True)
Calling processor.save_pretrained()
on this model renames
extra_special_tokens
to a non-standard model_specific_special_tokens
key and drops added_tokens_decoder
entirely ā a documented
transformers
round-trip bug (also seen on Phi-4-mini). Copy the files
directly from the original instead:
import shutil
from pathlib import Path
model_path = Path("./medgemma-1.5-4b-it")
quantized_model_path = Path(f"./{QUANTIZED_MODEL_ID}")
files_to_copy = [
"tokenizer.json",
"tokenizer_config.json",
"special_tokens_map.json",
"preprocessor_config.json",
"chat_template.json",
]
for fname in files_to_copy:
src = model_path / fname
if src.exists():
shutil.copy(src, quantized_model_path / fname)
Gemma3's newer nested rope_parameters
format (full_attention
/
sliding_attention
sub-dicts) isn't recognized by some downstream
tooling expecting a flat schema with a top-level rope_type
key. The
key existed before, just nested ā this restructures it, it doesn't add
something absent. If you're following along in a notebook, add
%%writefile patch_rope_config.py
as the first line of the cell to
write it straight to disk:
import json
import shutil
import sys
from pathlib import Path
def patch_config(config_path: str, dominant_rope_type: str = "default"):
config_path = Path(config_path)
backup_path = config_path.with_suffix(".json.bak")
if not backup_path.exists():
shutil.copy2(config_path, backup_path)
print(f"Backed up original to {backup_path}")
else:
print(f"Backup already exists at {backup_path}, not overwriting it")
with open(config_path, "r") as f:
config = json.load(f)
text_config = config.get("text_config")
if text_config is None:
print("ERROR: no 'text_config' block found ā is this the right file?")
sys.exit(1)
rope_params = text_config.get("rope_parameters")
if rope_params is None:
print("ERROR: no 'rope_parameters' found under text_config")
sys.exit(1)
if "rope_type" in rope_params:
print("'rope_type' key already present at top level ā nothing to do")
return
new_rope_params = {"rope_type": dominant_rope_type}
new_rope_params.update(rope_params)
text_config["rope_parameters"] = new_rope_params
with open(config_path, "w") as f:
json.dump(config, f, indent=2)
print(f"Patched {config_path}: added top-level rope_type='{dominant_rope_type}' "
f"inside rope_parameters (full_attention/sliding_attention left untouched)")
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python patch_rope_config.py /path/to/config.json [rope_type]")
sys.exit(1)
path = sys.argv[1]
rtype = sys.argv[2] if len(sys.argv) > 2 else "default"
patch_config(path, rtype)
python patch_rope_config.py ./medgemma-1.5-4b-it-W4A16-G128/config.json
Keeps a .json.bak
backup before touching anything, and is idempotent
ā safe to re-run if you're not sure whether it's already been applied.
A known llm-compressor
bug (issue #1546):
transformers
=4.52's multimodal refactor changed internal weight
naming during model , but llm-compressor
's
quantization_config.ignore
list is generated from the converted naming
scheme without being reconciled against the actual saved safetensors key
names. Left uncorrected, the ignore list matches no real tensor ā the
vision tower would silently get quantized despite the recipe saying
otherwise.
import json
def fix_model_config(config_path):
with open(config_path) as f:
config = json.load(f)
ignore_list = config["quantization_config"]["ignore"]
new_ignore = []
for entry in ignore_list:
if entry.startswith("model.vision_tower."):
fixed = entry.replace("model.vision_tower.", "vision_tower.vision_model.", 1)
new_ignore.append(fixed)
else:
new_ignore.append(entry) # e.g. "lm_head" stays as-is
config["quantization_config"]["ignore"] = new_ignore
with open(config_path, "w") as f:
json.dump(config, f, indent=2)
print("Sample before:", ignore_list[0])
print("Sample after: ", new_ignore[0])
fix_model_config("medgemma-1.5-4b-it-W4A16-G128/config.json")
If you set HF_TOKEN
in your .env
back in Step 2, huggingface_hub
picks it up automatically. Otherwise, hf auth login
first.
Don't use model.push_to_hub() ā it calls
save_pretrained()
againKeyError: 'weight'
(no plain weight
key left to compress a second
from huggingface_hub import HfApi
REPO_ID = f"<your-hf-username-or-org>/{QUANTIZED_MODEL_ID}"
api = HfApi()
api.create_repo(REPO_ID, exist_ok=True)
api.upload_folder(folder_path=QUANTIZED_MODEL_ID, repo_id=REPO_ID)
Result: 8.6 GB in BF16 -> 5.2 GB quantized. Vision tower,
multimodal projector, and lm_head
stay unquantized, which is why it's
not the full theoretical 75% reduction.
Model: confamnode/medgemma-1.5-4b-it-W4A16-G128
More technical blogs at ConfamNode