My MRI interpretation vision model pointed at the right kidney and wrote "left"

A developer building an MRI/CT interpretation tool found that left/right laterality errors were the most common failure across four vision models (Claude, Gemini, Grok, and Google's MedGemma) tested on eight real studies. The root cause is that models describe the side of the displayed image rather than the patient's anatomical side, since axial scans show the patient's right on the viewer's left. The developer added a model-free check that compares a finding's RAS world-x coordinate against the side word in the generated text, using an 8 mm midline dead zone and multilingual prefix regexes to catch mismatches.

I build a tool that reads MRI and CT scans with general vision models and explains the findings in plain language. The most repeated error in that pipeline isn't a missed tumour or an invented measurement. It's left and right . We ran eight real studies through four readers Claude, Gemini, Grok and Google's MedGemma , with the radiologist's signed report as ground truth. Here's what came back: In medicine a left/right swap is a "never event". You don't ship a report that does this, and a better prompt doesn't fix it. So here is where the error comes from, and the small, model-free check that now catches it. The detail that cracked it came from MedGemma. We ask each reader for two things per finding: a sentence, and a location on a specific slice. On one study MedGemma put its crosshair 17–26 mm to the right of the midline , which matched the radiologist. Then it wrote that the finding was "slightly to the left" . The crosshair was correct. The sentence described the side of the picture , not the side of the patient . That's the radiological convention catching the model out. An axial scan is displayed as if you're standing at the patient's feet looking up, so the patient's right is on the left of the screen. A model trained mostly on ordinary photos describes what it sees: "on the left of the image". In a report, that's the wrong side. This failure turns out to be useful. When the coordinates are right and only the sentence is wrong, you can check one against the other without involving a model at all. Our volumes are converted from DICOM to NIfTI in the browser dcm2niix compiled to WebAssembly . A NIfTI file carries an affine: a 4×4 matrix mapping voxel indices i, j, k to millimetres in world space. That space is RAS : +x is the patient's R ight, +y Anterior, +z Superior. So once a finding has a voxel location, the sign of its world-x gives a second, independent answer about which side it's on: export function voxToMm affine: Affine, i, j, k : number, number, number { const m = r: number = affine r 0 i + affine r 1 j + affine r 2 k + affine r 3 return m 0 , m 1 , m 2 as number, number, number } const OFF MIDLINE MM = 8 export function sideOfPoint mm: number, number, number : Side | null { const x = mm 0 if Number.isFinite x || Math.abs x < OFF MIDLINE MM return null return x 0 ? 'right' : 'left' // RAS: +x is the patient's RIGHT } The 8 mm dead zone matters. A central disc extrusion sits at x ≈ 0 and belongs to neither side, and floating-point noise shouldn't get read as a side. Reports come back in the user's language, so an English-only regex would silently check nothing for most users. The side words are prefix patterns, because almost every one of these languages inflects them destro/destra , rechts/rechten , prawy/prawa : js const RIGHT WORDS = /\bright\w |\bdestr aeio \w |\bderech\w |\bdroit\w |\brecht aensr \w |\bdireit\w |\bpraw ? d \w |\bsağ\w |يمين|أيمن|اليمنى/iu const LEFT WORDS = /\bleft\w |\bsinistr\w |\bizquierd\w |\bgauche\w |\blink aensr \w |\besquerd\w |\blew\w |\bsol\b|يسار|أيسر|اليسرى/iu const BOTH SIDES = /\bbilateral\w |\bbilaterale?\w |\bbeidseit\w |\bobustronn\w |\bambos\b|\bambas\b|\bdes deux côtés\b|ثنائي/iu Two traps are handled on purpose: \b needs a non-word character before the r , and b is a word character. MRI reports say "bright" constantly. ? d . Then there's the rule that took longest to get right: a sentence that names both sides claims neither. export function sideClaimed text: string : Side | null { if text return null if BOTH SIDES.test text return null const right = RIGHT WORDS.test text const left = LEFT WORDS.test text if right === left return null // none, or one of each return right ? 'right' : 'left' } "The left L5 root is displaced; the right is not" mentions both words. If you picked either one, you'd be inventing a claim the sentence never made. This is the part I got wrong first. A left knee MRI sits entirely at negative x. Every finding in it is "on the left", while the text is talking about the medial and lateral compartments of that one knee. Run the check there and every knee report gets flagged. So the sign of x only means a body side when the field of view reaches well past the midline in both directions. You can't read that off the origin, because the affine can rotate the grid. You have to check all eight corners: js const MIDLINE MARGIN MM = 40 function worldXRange affine: Affine, nx, ny, nz : number, number, number { let lo = Infinity, hi = -Infinity for const i of 0, nx - 1 for const j of 0, ny - 1 for const k of 0, nz - 1 { const x = voxToMm affine, i, j, k 0 lo = Math.min lo, x ; hi = Math.max hi, x } return lo, hi } export function straddlesMidline affine?: Affine, dims?: number, number, number { if affine || dims return false const lo, hi = worldXRange affine, dims return lo <= -MIDLINE MARGIN MM && hi = MIDLINE MARGIN MM } The 40 mm margin is generous on purpose. A spine or abdomen study clears it easily, and a limb never does. export function lateralityConflict opts: { text: string locations?: { mm?: number, number, number } affine?: Affine dims?: number, number, number } : Side | null { const claimed = sideClaimed opts.text if claimed return null if straddlesMidline opts.affine, opts.dims return null // Several points must agree with each other before they outvote the sentence. const sides = opts.locations ?? .map l = l.mm ? sideOfPoint l.mm : null .filter s : s is Side = s == null if sides.length === 0 || sides.every s = s === sides 0 return null return sides 0 === claimed ? null : sides 0 } Count the return null s. There are four ways for this function to say nothing, and one way for it to speak. That's deliberate. It isn't a classifier trying to arbitrate uncertain cases. It exists to catch the contradictions that are certain : the text names one side, the geometry clearly and consistently shows the other, and the scan covers both sides of the body. We don't silently rewrite the sentence. The crosshair is usually right that's the failure we measured , but the sentence is the part a person reads and acts on. So the conflict is stored on the finding sideConflict: 'right' and shown, instead of being quietly "fixed" by code that could itself be wrong. When a multimodal model gives you structured output next to prose , the two halves often fail independently. Coordinates, slice indices, bounding boxes and counts come from a different part of the model's behaviour than the sentence does. Whenever the domain gives you a deterministic way to relate the two here: an affine matrix and a coordinate convention , you get a check that costs microseconds, needs no second model, and can't hallucinate. It's worth finding those checks before reaching for another prompt. I'm Lorenzo, and I build ReadYourScan https://readyourscan.com , which opens DICOM/NIfTI studies in the browser and explains them in plain language. It's not a diagnosis. The browser viewer is open source MIT : lgdesignee/readyourscan-viewer https://github.com/lgdesignee/readyourscan-viewer .