Fine-tune our first 2B medical VLM on a single MacBook M4, beats Google's MedGemma 4B on MedXpertQA-MM eval dataset A developer's claim that their MadriMed-VL-2B model beats Google's MedGemma 4B on the MedXpertQA-MM benchmark is methodologically unsound because the evaluation likely ran text-only without loading images, yielding a 21.05% accuracy that is only 1.05 percentage points above random and within the 95% confidence interval of 19.3% to 22.8%. The developer should reclassify the result as a text-only baseline and rerun with proper multimodal validation before claiming superiority. It appears that the performance of the multimodal component may not have been evaluated: Methodology review: MadriMed-VL-2B vs MedGemma 4B on MedXpertQA-MM, SLAKE, and VQA-RAD This is a promising project, but I would be careful with the current headline. The compact-medical-VLM direction is genuinely interesting: fine-tuning a 2B vision-language model locally, making it usable on consumer hardware, and testing it on medical VQA / medical reasoning tasks is a worthwhile experiment. The useful contribution could be: a reproducible small-model medical VLM workflow that runs locally and is evaluated honestly . But the current benchmark claim: MadriMed-VL-2B beats Google’s MedGemma 4B on MedXpertQA-MM. is not methodologically safe yet . The short version is: The reported 21.05% MedXpertQA-MM score appears to be a text-only / zero-image multiple-choice run , not a confirmed multimodal MedXpertQA-MM score. That does not mean the model or project is bad. It means the benchmark result needs to be reclassified and rerun with stricter multimodal validation. Why I would not treat the current MedXpertQA-MM score as a confirmed multimodal result The key issue is an image-field mismatch. The MedXpertQA-MM evaluation loop appears to read: image list = row.get "image", But MedXpertQA-MM uses an images field for the multimodal image filenames, not a singular image field. This matters because row.get "image", does not fail loudly. It silently returns an empty list when the field is missing. Then the generation function can continue as a text-only run. That is exactly what the printed result suggests: MedXpertQA-MM Results n=2000 Accuracy : 21.05% 421/2000 Unknown : 0 0.0% Random Baseline: 20.00% Per-Image-Count Breakdown: 0 image s : 21.1% 421/2000 So all 2,000 MedXpertQA-MM examples appear to have been evaluated with 0 loaded images . That changes the interpretation completely. Instead of: MedXpertQA-MM multimodal score: 21.05%. I would report it as: MedXpertQA-MM text-only / zero-image baseline: 21.05%. Useful references: Why this matters statistically MedXpertQA-MM has 5 answer choices per question. A random baseline is therefore about: 20% The reported score is: 421 / 2000 = 21.05% That is only 1.05 percentage points above random . A rough 95% binomial confidence interval for 421 correct out of 2,000 is approximately: 19.3% to 22.8% That interval includes 20%. So even before the image-loading issue, 21.05% is not strong evidence of meaningful MedXpertQA-MM generalization. With the image-loading issue, the safer interpretation is: The current result is a near-random text-only MCQ baseline, not evidence that the model beat MedGemma on multimodal MedXpertQA-MM. This is also why I would avoid saying “beats MedGemma” until the corrected multimodal evaluation is rerun. For evaluation statistics and paired comparisons, see: Why the SLAKE / VQA-RAD gap is still meaningful The reported SLAKE and VQA-RAD numbers are below MedGemma: | Benchmark | MadriMed-VL-2B | MedGemma 4B | Gap | | SLAKE | 65.7 | 72.3 | -6.6 | | VQA-RAD | 43.09 | 49.9 | -6.81 | | MedXpertQA-MM | 21.05 | 18.8 | +2.25 | The SLAKE / VQA-RAD gap is plausible and informative. Those datasets stress short-answer medical visual grounding : - yes/no calibration; - modality recognition; - anatomy / organ recognition; - abnormality recognition; - concise answer formatting; - answer normalization; - tokenized F1 or exact-match scoring. A model can become more medically fluent or better at multiple-choice answer selection while still being weaker on short-answer visual grounding. References: But the MedXpertQA result probably did not “hold up despite the SLAKE/VQA-RAD gap” yet. The more likely explanation is simpler: SLAKE and VQA-RAD measured image-question answering, while the current MedXpertQA-MM run likely did not pass images at all. So I would not explain the MedXpertQA number as successful multimodal generalization yet. I would first fix and rerun the evaluation. Is the current methodology good? For an exploratory notebook: yes, it is useful . For a public benchmark claim against MedGemma: not yet . The main methodological problems are: | Issue | Why it matters | image vs images mismatch | Converts MedXpertQA-MM into a zero-image run | Silent fallback to | Hides dataset-schema bugs | Printed result shows 0 image s for all examples | Confirms the multimodal path probably failed | | 21.05% is near the 20% random baseline | Weak statistical evidence | | Local score compared to MedGemma model-card score | Not a same-harness comparison | | Different benchmark metrics | SLAKE/VQA-RAD are often tokenized-F1 style; MedXpertQA is accuracy | | No text-only / shuffled-image ablations | Cannot prove image use | | No confidence intervals / paired test | Cannot judge whether a small delta is meaningful | | No per-slice analysis | Cannot identify what actually improved | | No leakage checks described | Medical VQA datasets are small and reused | The strongest version of the work would fix these issues and then report results more conservatively. What I would do next 1. Reclassify the current 21.05% result I would change the result table from this: | Benchmark | MadriMed-VL-2B | MedGemma 4B | | MedXpertQA-MM | 21.05 | 18.8 | to this: | Benchmark | Mode | MadriMed-VL-2B | Interpretation | | MedXpertQA-MM | Text-only / zero-image run | 21.05 | Near-random 5-way MCQ baseline | | MedXpertQA-MM | Correct image + text | pending | True multimodal rerun | | MedXpertQA-MM | Shuffled image + text | pending | Image-use control | | MedXpertQA-MM | Options only | pending | Choice-prior control | | SLAKE | Image + text | 65.7 | Short-answer medical VQA | | VQA-RAD | Image + text | 43.09 | Radiology VQA | That framing is much safer. 2. Make image loading strict For MedXpertQA-MM, missing images should be a hard error. Do not use: image list = row.get "image", Use something stricter: python from pathlib import Path from PIL import Image def load medxpertqa mm example row, image root: Path : required = "id", "question", "options", "label", "images" missing = k for k in required if k not in row if missing: raise KeyError f"Missing required fields: {missing}. Available keys: {list row.keys }" image list = row "images" if isinstance image list, str : image list = image list if not isinstance image list, list : raise TypeError f"Expected row 'images' to be a list, got {type image list }" if len image list == 0: raise ValueError f"MedXpertQA-MM row has no image filenames: id={row 'id' }" images = for filename in image list: path = image root / filename if not path.exists : raise FileNotFoundError f"Missing image file: {path}" images.append Image.open path .convert "RGB" if len images = len image list : raise RuntimeError f"Image count mismatch for id={row 'id' }: " f"filenames={len image list }, loaded={len images }" return { "id": row "id" , "question": row "question" , "options": row "options" , "label": row "label" .strip .upper , "image filenames": image list, "images": images, "medical task": row.get "medical task" , "body system": row.get "body system" , "question type": row.get "question type" , } Before running inference, print image telemetry: python from collections import Counter filename counts = Counter loaded counts = Counter bad rows = for row in dataset: try: ex = load medxpertqa mm example row, IMAGE DIR filename counts len ex "image filenames" += 1 loaded counts len ex "images" += 1 except Exception as e: bad rows.append {"id": row.get "id" , "error": repr e } print "Rows:", len dataset print "Image filename counts:", filename counts print "Loaded image counts:", loaded counts print "Bad rows:", len bad rows assert len bad rows == 0 assert loaded counts 0 == 0 assert sum k v for k, v in loaded counts.items 0 Every multimodal result should include something like: | Run | Rows | Rows with image filenames | Rows with loaded images | Zero-image rows | | MedXpertQA-MM image+text | 2000 | 2000 | 2000 | 0 | If Zero-image rows = 2000 , it is not a multimodal evaluation. 3. Add required MedXpertQA-MM ablations For MedXpertQA-MM, one score is not enough. I would run at least four modes: | Mode | Images | Text | Purpose | | Text only | none | question + choices | Measures vignette / option-prior reasoning | | Correct image + text | correct images | question + choices | Intended multimodal evaluation | | Shuffled image + text | wrong images | question + choices | Tests whether correct images matter | | Options only | none | choices only | Tests answer-choice / label-position priors | Optional fifth mode: | Mode | Images | Text | Purpose | | Image + options only | correct images | minimal question + choices | Tests visual contribution without full vignette | How to interpret: | Pattern | Interpretation | | Correct image+text text-only | Images likely help | | Correct image+text shuffled-image | Correct images matter | | Shuffled-image ≈ correct image+text | Model may not use image content | | Text-only ≈ image+text | Benchmark or model is text-dominant | | Options-only above random | Answer-choice priors exist | | All modes near random | Model is not solving the benchmark | The key evidence for a real multimodal gain is: correct image + text text only correct image + text shuffled image + text Without that, I would not claim MedXpertQA-MM multimodal generalization. 4. Compare against the base 2B model first Before comparing to MedGemma, compare MadriMed-VL-2B to its own base model. The most important question is: Did fine-tuning actually improve the 2B model? Use a table like this: | Model | MedX text-only | MedX image+text | MedX shuffled | SLAKE | VQA-RAD | | Base 2B VLM | | | | | | | MadriMed-VL-2B | | | | | | | Delta | | | | | | Possible interpretations: | Pattern | Meaning | | Fine-tuned model improves text-only only | Better medical language / MCQ prior | | Fine-tuned model improves image+text over shuffled | Better visual grounding | | Fine-tuned model improves SLAKE/VQA-RAD | Better short-answer medical VQA | | Fine-tuned model improves MCQ but not SLAKE/VQA-RAD | Better answer-choice reasoning, weaker visual grounding | | No improvement over base | Fine-tune may not be effective | A strong result does not have to beat MedGemma immediately. A strong result can be: Fine-tuning improves a compact 2B VLM substantially over its base model, while remaining runnable locally. 5. Compare MedGemma only in the same harness Do not make the main claim by comparing a local notebook score to a model-card score. For a fair MedGemma comparison, run both models through the same evaluation harness: | Component | Requirement | | Dataset revision | Same for all models | | Split | Same for all models | | Image files | Same files | | Image loader | Same strict loader | | Prompt | Same task prompt, with only model-specific chat-template adaptation | | Decoding | Same deterministic settings | | Answer extractor | Same | | Metric | Same | | Unknown handling | Same | | Logs | Same JSONL schema | A fair comparison table: | Model | Mode | Accuracy | 95% CI | Unknown | Same harness | | Base 2B | Image+text | | | | yes | | MadriMed-VL-2B | Image+text | | | | yes | | MedGemma 4B | Image+text | | | | yes | Only after this should you say whether MadriMed beats MedGemma. Prompting suggestions MedXpertQA-MM prompt Use a strict MCQ prompt and short deterministic generation: You are answering a medical multiple-choice question. Use the clinical information and all provided images. Question: