{"slug": "revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med", "title": "Revolutionizing Dermatology: Building an Offline Skin Lesion Segmenter with Med-SAM and CoreML", "summary": "A developer built an offline skin lesion segmenter for mobile devices by optimizing the Med-SAM model with CoreML and ONNX Runtime. The approach enables real-time, privacy-preserving segmentation on iOS and cross-platform apps using React Native. The pipeline includes model distillation, quantization, and hardware acceleration to achieve medical-grade performance.", "body_md": "In the world of digital health, the gap between \"research-grade AI\" and \"production-ready mobile apps\" is often a chasm. When it comes to **dermatology AI**, precision is non-negotiable. Identifying a suspicious mole or a flare-up of dermatitis requires more than just a bounding box; it requires pixel-perfect edge detection.\n\nToday, we are diving deep into **Skin Lesion Screening Engineering**. We will explore how to bridge the gap by taking the powerful **Med-SAM** (Medical Segment Anything Model) and optimizing it for real-time, offline performance on mobile devices using **CoreML** and **ONNX Runtime**. By implementing **mobile image segmentation** and **edge-side inference**, we ensure user privacy while maintaining high-fidelity diagnostic assistance.\n\nScaling a transformer-based model like Med-SAM for a mobile environment requires a strategic pipeline. We can't just throw a 1GB model at a smartphone and expect 60FPS. We need a hybrid approach: distillation, quantization, and efficient hardware acceleration.\n\n``` php\ngraph TD\n    A[User Camera / Gallery] -->|Raw Image| B[Preprocessing & Resizing]\n    B --> C{Inference Engine}\n    C -->|iOS/Neural Engine| D[CoreML Model]\n    C -->|Cross-Platform| E[ONNX Runtime Mobile]\n    D --> F[Med-SAM Encoder/Decoder]\n    E --> F\n    F -->|Logits| G[Post-processing & Thresholding]\n    G -->|Segmentation Mask| H[UI Overlay: React Native]\n    H --> I[Final Assessment & Heatmap]\n```\n\nTo follow this advanced guide, you should be comfortable with:\n\nMed-SAM is a specialized variant of Meta’s Segment Anything Model, fine-tuned on massive medical datasets. However, the standard ViT-H (Vision Transformer) backbone is too heavy for mobile. We use a **Mobile-SAM** architecture but initialize it with Med-SAM's medical weights through knowledge distillation.\n\nFirst, we need to convert the PyTorch weights. Using `coremltools`\n\n, we can target the Apple Neural Engine (ANE).\n\n``` python\nimport torch\nimport coremltools as ct\nfrom med_sam import medsam_model_registry\n\n# 1. Load the fine-tuned Med-SAM model\nmodel = medsam_model_registry[\"vit_b\"](checkpoint=\"medsam_vit_b.pth\")\nmodel.eval()\n\n# 2. Trace the model with a dummy input\nexample_input = torch.rand(1, 3, 1024, 1024) \ntraced_model = torch.jit.trace(model, example_input)\n\n# 3. Convert to CoreML (Optimize for ANE)\nmlmodel = ct.convert(\n    traced_model,\n    inputs=[ct.TensorType(shape=example_input.shape)],\n    compute_units=ct.ComputeUnit.ALL, # Utilize GPU and Neural Engine\n    minimum_deployment_target=ct.target.iOS16\n)\n\nmlmodel.save(\"SkinMedSAM.mlpackage\")\n```\n\nWhile the \"brain\" of our app is the model, the \"body\" is **React Native**. For high-performance vision tasks, we use `react-native-vision-camera`\n\ncombined with a custom frame processor or the **ONNX Runtime React Native** library.\n\n``` js\nimport { OrtClient, Tensor } from 'onnxruntime-react-native';\n\nconst runSegmentation = async (imagePath: string) => {\n  try {\n    // Load the model\n    const session = await OrtClient.create('skin_medsam_quantized.onnx');\n\n    // Convert image to Tensor (Logic for resizing to 1024x1024)\n    const inputTensor = await preprocessImageToTensor(imagePath);\n\n    // Run Inference\n    const outputs = await session.run({ \"input\": inputTensor });\n    const maskData = outputs[\"output\"].data;\n\n    // Process mask to display on UI\n    return maskData;\n  } catch (e) {\n    console.error(\"Inference failed\", e);\n  }\n};\n```\n\nBuilding a prototype is easy, but making it \"medical-grade\" requires rigorous attention to lighting conditions, skin tone diversity, and latency.\n\nFor advanced architectural patterns, such as implementing **streaming inference** or **federated learning** to improve your skin lesion models without compromising user data, I highly recommend checking out the technical deep-dives at [ WellAlly Blog](https://www.wellally.tech/blog). They provide extensive resources on productionizing vision models and navigating the complexities of healthcare AI engineering.\n\nOnce we get the segmentation mask, we need to overlay it on the camera feed. This is where we calculate the **area** of the lesion and its **regularity**—key features for early screening.\n\n``` js\nimport { Canvas, Path, Skia } from \"@shopify/react-native-skia\";\n\n// Drawing the mask over the lesion\nconst LesionMask = ({ points }) => {\n  const path = Skia.Path.Make();\n  // Simplified logic to convert model output to SVG path\n  path.moveTo(points[0].x, points[0].y);\n  points.forEach(p => path.lineTo(p.x, p.y));\n  path.close();\n\n  return (\n    <Canvas style={{ flex: 1 }}>\n      <Path path={path} color=\"rgba(255, 0, 0, 0.4)\" style=\"fill\" />\n    </Canvas>\n  );\n};\n```\n\nEngineering a skin lesion screening tool is a perfect example of \"Learning in Public.\" By combining **Med-SAM**'s specialized medical knowledge with **CoreML**'s hardware acceleration, we can create life-saving tools that live right in our pockets.\n\n**Key Takeaways:**\n\nHave you tried deploying Vision Transformers to mobile? What was your biggest hurdle? Let's discuss in the comments! 👇\n\n*If you're looking for more production-ready examples and deep architectural insights, don't forget to visit wellally.tech/blog for the latest in AI and Healthcare engineering.* 💻✨", "url": "https://wpnews.pro/news/revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med", "canonical_source": "https://dev.to/beck_moulton/revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med-sam-and-coreml-4hcl", "published_at": "2026-06-16 00:08:00+00:00", "updated_at": "2026-06-16 00:47:45.753790+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision"], "entities": ["Med-SAM", "CoreML", "ONNX Runtime", "React Native", "Apple Neural Engine", "WellAlly", "Meta", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med", "markdown": "https://wpnews.pro/news/revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med.md", "text": "https://wpnews.pro/news/revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med.txt", "jsonld": "https://wpnews.pro/news/revolutionizing-dermatology-building-an-offline-skin-lesion-segmenter-with-med.jsonld"}}