{"slug": "fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-google-s-4b-on", "title": "Fine-tune our first 2B medical VLM on a single MacBook M4, beats Google's MedGemma 4B on MedXpertQA-MM eval dataset", "summary": "A developer's claim that their MadriMed-VL-2B model beats Google's MedGemma 4B on the MedXpertQA-MM benchmark is methodologically unsound because the evaluation likely ran text-only without loading images, yielding a 21.05% accuracy that is only 1.05 percentage points above random and within the 95% confidence interval of 19.3% to 22.8%. The developer should reclassify the result as a text-only baseline and rerun with proper multimodal validation before claiming superiority.", "body_md": "It appears that the performance of the multimodal component may not have been evaluated:\n\nMethodology review: MadriMed-VL-2B vs MedGemma 4B on MedXpertQA-MM, SLAKE, and VQA-RAD\n\nThis is a promising project, but I would be careful with the current headline.\n\nThe compact-medical-VLM direction is genuinely interesting: fine-tuning a 2B vision-language model locally, making it usable on consumer hardware, and testing it on medical VQA / medical reasoning tasks is a worthwhile experiment. The useful contribution could be: **a reproducible small-model medical VLM workflow that runs locally and is evaluated honestly**.\n\nBut the current benchmark claim:\n\nMadriMed-VL-2B beats Google’s MedGemma 4B on MedXpertQA-MM.\n\nis **not methodologically safe yet**.\n\nThe short version is:\n\nThe reported **21.05% MedXpertQA-MM score appears to be a text-only / zero-image multiple-choice run**, not a confirmed multimodal MedXpertQA-MM score.\n\nThat does **not** mean the model or project is bad. It means the benchmark result needs to be reclassified and rerun with stricter multimodal validation.\n\nWhy I would not treat the current MedXpertQA-MM score as a confirmed multimodal result\n\nThe key issue is an image-field mismatch.\n\nThe MedXpertQA-MM evaluation loop appears to read:\n\n```\nimage_list = row.get(\"image\", [])\n```\n\nBut MedXpertQA-MM uses an `images`\n\nfield for the multimodal image filenames, not a singular `image`\n\nfield.\n\nThis matters because `row.get(\"image\", [])`\n\ndoes not fail loudly. It silently returns an empty list when the field is missing. Then the generation function can continue as a text-only run.\n\nThat is exactly what the printed result suggests:\n\n```\nMedXpertQA-MM Results (n=2000)\nAccuracy      : 21.05% (421/2000)\nUnknown       : 0 (0.0%)\nRandom Baseline: 20.00%\n\nPer-Image-Count Breakdown:\n0 image(s): 21.1% (421/2000)\n```\n\nSo all 2,000 MedXpertQA-MM examples appear to have been evaluated with **0 loaded images**.\n\nThat changes the interpretation completely.\n\nInstead of:\n\nMedXpertQA-MM multimodal score: 21.05%.\n\nI would report it as:\n\nMedXpertQA-MM text-only / zero-image baseline: 21.05%.\n\nUseful references:\n\nWhy this matters statistically\n\nMedXpertQA-MM has 5 answer choices per question. A random baseline is therefore about:\n\n```\n20%\n```\n\nThe reported score is:\n\n```\n421 / 2000 = 21.05%\n```\n\nThat is only **1.05 percentage points above random**.\n\nA rough 95% binomial confidence interval for 421 correct out of 2,000 is approximately:\n\n```\n19.3% to 22.8%\n```\n\nThat interval includes 20%. So even before the image-loading issue, 21.05% is not strong evidence of meaningful MedXpertQA-MM generalization.\n\nWith the image-loading issue, the safer interpretation is:\n\nThe current result is a near-random text-only MCQ baseline, not evidence that the model beat MedGemma on multimodal MedXpertQA-MM.\n\nThis is also why I would avoid saying “beats MedGemma” until the corrected multimodal evaluation is rerun.\n\nFor evaluation statistics and paired comparisons, see:\n\nWhy the SLAKE / VQA-RAD gap is still meaningful\n\nThe reported SLAKE and VQA-RAD numbers are below MedGemma:\n\n| Benchmark |\nMadriMed-VL-2B |\nMedGemma 4B |\nGap |\n| SLAKE |\n65.7 |\n72.3 |\n-6.6 |\n| VQA-RAD |\n43.09 |\n49.9 |\n-6.81 |\n| MedXpertQA-MM |\n21.05 |\n18.8 |\n+2.25 |\n\nThe SLAKE / VQA-RAD gap is plausible and informative.\n\nThose datasets stress **short-answer medical visual grounding**:\n\n- yes/no calibration;\n- modality recognition;\n- anatomy / organ recognition;\n- abnormality recognition;\n- concise answer formatting;\n- answer normalization;\n- tokenized F1 or exact-match scoring.\n\nA model can become more medically fluent or better at multiple-choice answer selection while still being weaker on short-answer visual grounding.\n\nReferences:\n\nBut the MedXpertQA result probably did **not** “hold up despite the SLAKE/VQA-RAD gap” yet. The more likely explanation is simpler:\n\nSLAKE and VQA-RAD measured image-question answering, while the current MedXpertQA-MM run likely did not pass images at all.\n\nSo I would not explain the MedXpertQA number as successful multimodal generalization yet. I would first fix and rerun the evaluation.\n\nIs the current methodology good?\n\nFor an exploratory notebook: **yes, it is useful**.\n\nFor a public benchmark claim against MedGemma: **not yet**.\n\nThe main methodological problems are:\n\n| Issue |\nWhy it matters |\n`image` vs `images` mismatch |\nConverts MedXpertQA-MM into a zero-image run |\nSilent fallback to `[]` |\nHides dataset-schema bugs |\nPrinted result shows `0 image(s)` for all examples |\nConfirms the multimodal path probably failed |\n| 21.05% is near the 20% random baseline |\nWeak statistical evidence |\n| Local score compared to MedGemma model-card score |\nNot a same-harness comparison |\n| Different benchmark metrics |\nSLAKE/VQA-RAD are often tokenized-F1 style; MedXpertQA is accuracy |\n| No text-only / shuffled-image ablations |\nCannot prove image use |\n| No confidence intervals / paired test |\nCannot judge whether a small delta is meaningful |\n| No per-slice analysis |\nCannot identify what actually improved |\n| No leakage checks described |\nMedical VQA datasets are small and reused |\n\nThe strongest version of the work would fix these issues and then report results more conservatively.\n\nWhat I would do next\n\n1. Reclassify the current 21.05% result\n\nI would change the result table from this:\n\n| Benchmark |\nMadriMed-VL-2B |\nMedGemma 4B |\n| MedXpertQA-MM |\n21.05 |\n18.8 |\n\nto this:\n\n| Benchmark |\nMode |\nMadriMed-VL-2B |\nInterpretation |\n| MedXpertQA-MM |\nText-only / zero-image run |\n21.05 |\nNear-random 5-way MCQ baseline |\n| MedXpertQA-MM |\nCorrect image + text |\npending |\nTrue multimodal rerun |\n| MedXpertQA-MM |\nShuffled image + text |\npending |\nImage-use control |\n| MedXpertQA-MM |\nOptions only |\npending |\nChoice-prior control |\n| SLAKE |\nImage + text |\n65.7 |\nShort-answer medical VQA |\n| VQA-RAD |\nImage + text |\n43.09 |\nRadiology VQA |\n\nThat framing is much safer.\n\n2. Make image loading strict\n\nFor MedXpertQA-MM, missing images should be a hard error.\n\nDo not use:\n\n```\nimage_list = row.get(\"image\", [])\n```\n\nUse something stricter:\n\n``` python\nfrom pathlib import Path\nfrom PIL import Image\n\ndef load_medxpertqa_mm_example(row, image_root: Path):\n    required = [\"id\", \"question\", \"options\", \"label\", \"images\"]\n    missing = [k for k in required if k not in row]\n    if missing:\n        raise KeyError(f\"Missing required fields: {missing}. Available keys: {list(row.keys())}\")\n\n    image_list = row[\"images\"]\n\n    if isinstance(image_list, str):\n        image_list = [image_list]\n\n    if not isinstance(image_list, list):\n        raise TypeError(f\"Expected row['images'] to be a list, got {type(image_list)}\")\n\n    if len(image_list) == 0:\n        raise ValueError(f\"MedXpertQA-MM row has no image filenames: id={row['id']}\")\n\n    images = []\n    for filename in image_list:\n        path = image_root / filename\n        if not path.exists():\n            raise FileNotFoundError(f\"Missing image file: {path}\")\n\n        images.append(Image.open(path).convert(\"RGB\"))\n\n    if len(images) != len(image_list):\n        raise RuntimeError(\n            f\"Image count mismatch for id={row['id']}: \"\n            f\"filenames={len(image_list)}, loaded={len(images)}\"\n        )\n\n    return {\n        \"id\": row[\"id\"],\n        \"question\": row[\"question\"],\n        \"options\": row[\"options\"],\n        \"label\": row[\"label\"].strip().upper(),\n        \"image_filenames\": image_list,\n        \"images\": images,\n        \"medical_task\": row.get(\"medical_task\"),\n        \"body_system\": row.get(\"body_system\"),\n        \"question_type\": row.get(\"question_type\"),\n    }\n```\n\nBefore running inference, print image telemetry:\n\n``` python\nfrom collections import Counter\n\nfilename_counts = Counter()\nloaded_counts = Counter()\nbad_rows = []\n\nfor row in dataset:\n    try:\n        ex = load_medxpertqa_mm_example(row, IMAGE_DIR)\n        filename_counts[len(ex[\"image_filenames\"])] += 1\n        loaded_counts[len(ex[\"images\"])] += 1\n    except Exception as e:\n        bad_rows.append({\"id\": row.get(\"id\"), \"error\": repr(e)})\n\nprint(\"Rows:\", len(dataset))\nprint(\"Image filename counts:\", filename_counts)\nprint(\"Loaded image counts:\", loaded_counts)\nprint(\"Bad rows:\", len(bad_rows))\n\nassert len(bad_rows) == 0\nassert loaded_counts[0] == 0\nassert sum(k * v for k, v in loaded_counts.items()) > 0\n```\n\nEvery multimodal result should include something like:\n\n| Run |\nRows |\nRows with image filenames |\nRows with loaded images |\nZero-image rows |\n| MedXpertQA-MM image+text |\n2000 |\n2000 |\n2000 |\n0 |\n\nIf `Zero-image rows = 2000`\n\n, it is not a multimodal evaluation.\n\n3. Add required MedXpertQA-MM ablations\n\nFor MedXpertQA-MM, one score is not enough.\n\nI would run at least four modes:\n\n| Mode |\nImages |\nText |\nPurpose |\n| Text only |\nnone |\nquestion + choices |\nMeasures vignette / option-prior reasoning |\n| Correct image + text |\ncorrect images |\nquestion + choices |\nIntended multimodal evaluation |\n| Shuffled image + text |\nwrong images |\nquestion + choices |\nTests whether correct images matter |\n| Options only |\nnone |\nchoices only |\nTests answer-choice / label-position priors |\n\nOptional fifth mode:\n\n| Mode |\nImages |\nText |\nPurpose |\n| Image + options only |\ncorrect images |\nminimal question + choices |\nTests visual contribution without full vignette |\n\nHow to interpret:\n\n| Pattern |\nInterpretation |\n| Correct image+text > text-only |\nImages likely help |\n| Correct image+text > shuffled-image |\nCorrect images matter |\n| Shuffled-image ≈ correct image+text |\nModel may not use image content |\n| Text-only ≈ image+text |\nBenchmark or model is text-dominant |\n| Options-only above random |\nAnswer-choice priors exist |\n| All modes near random |\nModel is not solving the benchmark |\n\nThe key evidence for a real multimodal gain is:\n\n```\ncorrect image + text > text only\ncorrect image + text > shuffled image + text\n```\n\nWithout that, I would not claim MedXpertQA-MM multimodal generalization.\n\n4. Compare against the base 2B model first\n\nBefore comparing to MedGemma, compare MadriMed-VL-2B to its own base model.\n\nThe most important question is:\n\nDid fine-tuning actually improve the 2B model?\n\nUse a table like this:\n\n| Model |\nMedX text-only |\nMedX image+text |\nMedX shuffled |\nSLAKE |\nVQA-RAD |\n| Base 2B VLM |\n|\n|\n|\n|\n|\n| MadriMed-VL-2B |\n|\n|\n|\n|\n|\n| Delta |\n|\n|\n|\n|\n|\n\nPossible interpretations:\n\n| Pattern |\nMeaning |\n| Fine-tuned model improves text-only only |\nBetter medical language / MCQ prior |\n| Fine-tuned model improves image+text over shuffled |\nBetter visual grounding |\n| Fine-tuned model improves SLAKE/VQA-RAD |\nBetter short-answer medical VQA |\n| Fine-tuned model improves MCQ but not SLAKE/VQA-RAD |\nBetter answer-choice reasoning, weaker visual grounding |\n| No improvement over base |\nFine-tune may not be effective |\n\nA strong result does not have to beat MedGemma immediately. A strong result can be:\n\nFine-tuning improves a compact 2B VLM substantially over its base model, while remaining runnable locally.\n\n5. Compare MedGemma only in the same harness\n\nDo not make the main claim by comparing a local notebook score to a model-card score.\n\nFor a fair MedGemma comparison, run both models through the same evaluation harness:\n\n| Component |\nRequirement |\n| Dataset revision |\nSame for all models |\n| Split |\nSame for all models |\n| Image files |\nSame files |\n| Image loader |\nSame strict loader |\n| Prompt |\nSame task prompt, with only model-specific chat-template adaptation |\n| Decoding |\nSame deterministic settings |\n| Answer extractor |\nSame |\n| Metric |\nSame |\n| Unknown handling |\nSame |\n| Logs |\nSame JSONL schema |\n\nA fair comparison table:\n\n| Model |\nMode |\nAccuracy |\n95% CI |\nUnknown |\nSame harness |\n| Base 2B |\nImage+text |\n|\n|\n|\nyes |\n| MadriMed-VL-2B |\nImage+text |\n|\n|\n|\nyes |\n| MedGemma 4B |\nImage+text |\n|\n|\n|\nyes |\n\nOnly after this should you say whether MadriMed beats MedGemma.\n\nPrompting suggestions\n\nMedXpertQA-MM prompt\n\nUse a strict MCQ prompt and short deterministic generation:\n\n```\nYou are answering a medical multiple-choice question.\nUse the clinical information and all provided images.\n\nQuestion:\n<question>\n\nAnswer choices:\nA. <option_a>\nB. <option_b>\nC. <option_c>\nD. <option_d>\nE. <option_e>\n\nReturn only one letter: A, B, C, D, or E.\nFinal answer:\n```\n\nRecommended generation:\n\n```\ngenerate_kwargs = {\n    \"max_new_tokens\": 8,\n    \"do_sample\": False,\n}\n```\n\nIf the task only needs a letter, `max_new_tokens=512`\n\nis unnecessarily long and can increase answer-extraction noise.\n\nSLAKE / VQA-RAD prompt\n\nUse a short-answer prompt:\n\n```\nAnswer the medical image question using a short answer.\n\nFor yes/no questions, answer only yes or no.\nFor modality questions, answer only the modality name.\nFor anatomy questions, answer only the anatomical structure.\n\nQuestion:\n<question>\n\nAnswer:\n```\n\nRecommended generation:\n\n```\ngenerate_kwargs = {\n    \"max_new_tokens\": 16,\n    \"do_sample\": False,\n}\n```\n\nWhy different prompts?\n\nBecause MedXpertQA-MM is a multiple-choice reasoning task, while SLAKE and VQA-RAD are short-answer VQA tasks. A prompt that helps one can hurt the other.\n\nAnswer extraction suggestions\n\nFor MedXpertQA-MM:\n\n``` php\nimport re\n\ndef extract_mcq_letter(text: str) -> str:\n    text = text.strip().upper()\n\n    patterns = [\n        r\"FINAL\\s+ANSWER\\s*[:\\-]?\\s*\\(?([ABCDE])\\)?\",\n        r\"ANSWER\\s*[:\\-]?\\s*\\(?([ABCDE])\\)?\",\n        r\"^\\(?([ABCDE])\\)?[\\.\\)]?$\",\n        r\"\\b([ABCDE])\\b\",\n    ]\n\n    for pattern in patterns:\n        match = re.search(pattern, text)\n        if match:\n            return match.group(1)\n\n    return \"UNKNOWN\"\n```\n\nReport:\n\n| Diagnostic |\nWhy |\n| Accuracy counting unknown as wrong |\nConservative score |\n| Unknown rate |\nOutput-format compliance |\n| Answer distribution |\nDetects A/B/C/D/E bias |\n| Raw outputs |\nLets others audit extraction |\n| Per-choice accuracy |\nDetects answer-position artifacts |\n\nFor SLAKE / VQA-RAD, report both:\n\n- normalized exact match;\n- tokenized F1;\n- open-ended score;\n- closed-ended score;\n- yes/no accuracy.\n\nAdd answer-choice rotation for MedXpertQA-MM\n\nMultiple-choice models can exploit option-position bias. Rotate options.\n\n| Variant |\nOption order |\n| Original |\nA B C D E |\n| Rotation 1 |\nB C D E A |\n| Rotation 2 |\nC D E A B |\n| Rotation 3 |\nD E A B C |\n| Rotation 4 |\nE A B C D |\n\nThen map the predicted letter back to the semantic answer.\n\nReport:\n\n| Metric |\nMeaning |\n| Original-order accuracy |\nStandard score |\n| Rotation-mean accuracy |\nMore robust score |\n| Semantic consistency |\nWhether the same answer is chosen under rotations |\n| Letter bias |\nWhether the model over-picks A/B/C/D/E |\n\nIf accuracy collapses under rotation, the model may be exploiting option position rather than doing robust reasoning.\n\nAdd confidence intervals and paired tests\n\nFor accuracy:\n\n``` python\nfrom statsmodels.stats.proportion import proportion_confint\n\ndef wilson_ci(correct, total, alpha=0.05):\n    return proportion_confint(correct, total, alpha=alpha, method=\"wilson\")\n```\n\nFor model comparisons:\n\n``` python\nimport numpy as np\n\ndef paired_bootstrap_diff(a_correct, b_correct, n_boot=10000, seed=0):\n    rng = np.random.default_rng(seed)\n\n    a_correct = np.asarray(a_correct, dtype=np.float32)\n    b_correct = np.asarray(b_correct, dtype=np.float32)\n\n    assert len(a_correct) == len(b_correct)\n\n    n = len(a_correct)\n    diffs = []\n\n    for _ in range(n_boot):\n        idx = rng.integers(0, n, size=n)\n        diffs.append(a_correct[idx].mean() - b_correct[idx].mean())\n\n    return np.percentile(diffs, [2.5, 50, 97.5])\n```\n\nReport deltas:\n\n| Comparison |\nDelta |\n95% CI |\nMeaning |\n| MadriMed image+text − MadriMed text-only |\n|\n|\nVisual contribution |\n| MadriMed image+text − MadriMed shuffled-image |\n|\n|\nCorrect-image contribution |\n| MadriMed image+text − Base 2B image+text |\n|\n|\nFine-tuning gain |\n| MadriMed image+text − MedGemma image+text |\n|\n|\nExternal comparison |\n\nIf the confidence interval crosses zero, do not call it a clear win.\n\nAdd per-slice analysis\n\nMedXpertQA includes metadata such as medical task, body system, and question type. Use it.\n\nMedXpertQA-MM slice table\n\n| Slice |\nn |\nText-only |\nImage+text |\nShuffled-image |\nImage gain |\n| Diagnosis |\n|\n|\n|\n|\n|\n| Treatment |\n|\n|\n|\n|\n|\n| Basic medicine |\n|\n|\n|\n|\n|\n| Reasoning |\n|\n|\n|\n|\n|\n| Understanding |\n|\n|\n|\n|\n|\n| Cardiovascular |\n|\n|\n|\n|\n|\n| Dermatology-related |\n|\n|\n|\n|\n|\n| Radiology-heavy |\n|\n|\n|\n|\n|\n\nSLAKE / VQA-RAD slice table\n\n| Slice |\nScore |\nWhy it matters |\n| Open-ended |\n|\nHarder answer normalization |\n| Closed-ended |\n|\nOften yes/no-heavy |\n| Yes/no |\n|\nDetects yes/no bias |\n| Modality |\n|\nBasic visual recognition |\n| Organ/body part |\n|\nAnatomical grounding |\n| Abnormality |\n|\nClinical visual interpretation |\n| Location |\n|\nSpatial reasoning |\n\nThis will show whether the model is weak because it cannot see, cannot reason, cannot answer concisely, or cannot handle a particular modality.\n\nCheck for leakage\n\nMedical VQA datasets are small and frequently reused. Leakage checks are important.\n\nCheck:\n\n- exact image duplicates between train and test;\n- perceptual image duplicates;\n- repeated clinical vignettes;\n- repeated question/answer pairs;\n- synthetic data generated from benchmark examples;\n- captions or filenames that reveal labels.\n\nExample image hash check:\n\n``` python\nfrom PIL import Image\nimport imagehash\n\ndef phash(path):\n    return imagehash.phash(Image.open(path).convert(\"RGB\"))\n```\n\nThen compare training images against MedXpertQA-MM, SLAKE, and VQA-RAD evaluation images.\n\nReferences:\n\nTraining suggestions for the next version\n\nOnce evaluation is fixed, I would train in stages rather than mixing everything together.\n\nStage 1 — visual-medical grounding\n\nGoal: teach the model to see medical images.\n\nExamples:\n\n```\nQuestion: What modality is this?\nAnswer: CT\n\nQuestion: Which body region is shown?\nAnswer: chest\n\nQuestion: Does the image contain liver?\nAnswer: no\n\nQuestion: Is there pleural effusion?\nAnswer: yes\n```\n\nTrain on:\n\n- modality recognition;\n- anatomy recognition;\n- body-region recognition;\n- view / plane recognition;\n- presence / absence;\n- normal / abnormal.\n\nStage 2 — short-answer VQA\n\nGoal: improve SLAKE / VQA-RAD.\n\nExamples:\n\n```\nQuestion: Is there cardiomegaly?\nAnswer: yes\n\nQuestion: What organ is shown?\nAnswer: lung\n\nQuestion: What imaging modality is used?\nAnswer: x-ray\n```\n\nMake the answer short and canonical.\n\nStage 3 — clinical MCQ reasoning\n\nGoal: improve MedXpertQA-style reasoning.\n\nExample:\n\n```\nQuestion:\n<clinical_vignette_plus_image_context>\n\nAnswer choices:\nA. <option_a>\nB. <option_b>\nC. <option_c>\nD. <option_d>\nE. <option_e>\n\nFinal answer: C\n```\n\nStage 4 — mixed replay\n\nGoal: avoid overfitting to one format.\n\nMix:\n\n- short-answer VQA;\n- yes/no;\n- modality/anatomy;\n- MCQ;\n- general VLM instruction samples;\n- negative examples.\n\nFine-tuning target suggestions\n\nDo not only test one fine-tuning configuration.\n\nFor a VLM, language-only tuning can improve medical wording without improving image use. Test at least three variants:\n\n| Run |\nTuned components |\nWhat it tests |\n| A |\nLanguage layers only |\nBetter medical language / answer style |\n| B |\nProjector + language layers |\nBetter image-text alignment |\n| C |\nLate vision blocks + projector + language |\nBetter medical visual adaptation |\n\nThen evaluate:\n\n| Signal |\nWhat you want |\n| Correct-image vs text-only |\nImages help |\n| Correct-image vs shuffled-image |\nCorrect images matter |\n| SLAKE/VQA-RAD improvement |\nBetter visual grounding |\n| MCQ improvement |\nBetter clinical answer-choice reasoning |\n\nThe best run is not necessarily the one with the highest raw score. It is the one that improves the intended\n\nFinal verdict\n\nIs this good methodology?\n\nNot yet.\n\nIt is a useful exploratory experiment, but not a strong benchmark methodology for claiming a win over MedGemma.\n\nDid MedXpertQA generalization hold up?\n\nProbably not yet. The current MedXpertQA-MM result appears to be a zero-image / text-only run, and the score is near the 20% random baseline for a 5-way MCQ task.\n\nWhy are SLAKE / VQA-RAD lower?\n\nThat part is plausible. SLAKE and VQA-RAD stress short-answer visual grounding, modality/anatomy recognition, yes/no calibration, and answer normalization. A small model can lag there even if it becomes better at medical wording or MCQ-style output.\n\nWhat should happen next?\n\n- Fix\n`image`\n\n→ `images`\n\n.\n- Make missing images a hard error.\n- Treat 21.05% as a text-only baseline.\n- Rerun MedXpertQA-MM as image+text.\n- Add text-only, shuffled-image, and options-only ablations.\n- Run the base 2B model.\n- Run MedGemma in the same harness.\n- Report confidence intervals and paired deltas.\n- Break results down by task, body system, question type, open/closed type, and yes/no type.\n- Update the model card with conservative wording.\n\nShort summary\n\n- The project direction is good.\n- The current MedXpertQA-MM headline is not yet supported.\n- The evaluation appears to load\n`image`\n\n, but the dataset uses `images`\n\n.\n- The printed result shows every MedXpertQA-MM example under\n**0 image(s)**.\n- Therefore,\n**21.05% is best treated as a text-only / zero-image baseline**, not a multimodal score.\n- SLAKE / VQA-RAD gaps are plausible because those benchmarks test short-answer visual grounding.\n- The next step is a strict rerun with image validation, ablations, same-harness MedGemma comparison, and confidence intervals.\n\nUseful links:", "url": "https://wpnews.pro/news/fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-google-s-4b-on", "canonical_source": "https://discuss.huggingface.co/t/fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-googles-medgemma-4b-on-medxpertqa-mm-eval-dataset/176061#post_3", "published_at": "2026-08-18 13:06:15+00:00", "updated_at": "2026-08-18 13:13:49.991903+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-ethics"], "entities": ["MadriMed-VL-2B", "Google", "MedGemma 4B", "MedXpertQA-MM", "SLAKE", "VQA-RAD"], "alternates": {"html": "https://wpnews.pro/news/fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-google-s-4b-on", "markdown": "https://wpnews.pro/news/fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-google-s-4b-on.md", "text": "https://wpnews.pro/news/fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-google-s-4b-on.txt", "jsonld": "https://wpnews.pro/news/fine-tune-our-first-2b-medical-vlm-on-a-single-macbook-m4-beats-google-s-4b-on.jsonld"}}