{"slug": "bringing-the-monai-model-zoo-to-amd-instincttm-gpus-validated-inference-on-rocm", "title": "Bringing the MONAI Model Zoo to AMD Instinct™ GPUs: Validated Inference on ROCm", "summary": "AMD released Early Access validated, optimized inference support for five MONAI Model Zoo bundles on AMD Instinct GPUs as part of its Life Science 26.08 medical-imaging stack. The five bundles — Vista3D, SwinUNETR (BTCV), wholeBody_ct_segmentation, pancreas_ct_dints_segmentation and spleen_deepedit_annotation — run through the standard monai.bundle runner and clear their upstream Dice thresholds with unchanged weights and inference logic, using a ROCm overlay that applies channels-last 3D, BF16, torch.compile() and targeted fixes. Per-bundle compiled-PyTorch latencies range from 3.60 ms (spleen_deepedit) to 3,287 ms (pancreas_dints) on an AMD Instinct MI355X (gfx950) with ROCm 10.0, measured on the spleen_10.nii.gz CT volume.", "body_md": "# Bringing the MONAI Model Zoo to AMD Instinct™ GPUs: Validated Inference on ROCm[#](#bringing-the-monai-model-zoo-to-amd-instinct-gpus-validated-inference-on-rocm)\n\nThe Medical Open Network for AI (MONAI) Model Zoo is often the first stop when a researcher is building a medical imaging application: a shared shelf of ready-to-use, peer-reviewed models, where you can find a whole-body computed tomography (CT) segmenter or an interactive annotation network already trained, packaged, and ready to run.\n\nIn this blog you will learn how five of those bundles run validated, optimized inference on AMD\nInstinct™ GPUs, what the ROCm™ overlay does to get them there, and how to run them yourself. This\npost is one of a series on the AMD Life Science 26.08 medical-imaging stack. The companion posts\ncover [the hipCIM image-I/O library beneath it](https://rocm.blogs.amd.com/software-tools-optimization/hipcim-26-08/README.html) and\n[MONAI data loading on top of it](https://rocm.blogs.amd.com/artificial-intelligence/monai-hipcim-2608/README.html), with interactive annotation still to come.\n\nEach entry in the Model Zoo is a self-contained MONAI bundle with the model definition, the pre-trained weights, and a configuration-driven inference pipeline that can be launched with a single command.\n\nWith [AMD Life Science 26.08](https://rocm.docs.amd.com/projects/lifescience/en/latest/), AMD is releasing validated, optimized\ninference support for five MONAI Model Zoo bundles on AMD Instinct™ GPUs. This is an Early Access release focused on inference.\n\n## Optimized Inference Through a ROCm Overlay[#](#optimized-inference-through-a-rocm-overlay)\n\nFive Model Zoo bundles run optimized inference on AMD Instinct GPUs\nusing ordinary PyTorch, with no offline engine compilation required. The optimization arrives through a ROCm™ overlay. The overlay applies channels-last 3D, 16-bit brain floating point (BF16), `torch.compile()`, and targeted fixes on top of the stock bundle configuration.\n\nTable 1 lists the five bundles.\n\n| Bundle | Task | Architecture | Anatomy | \n|---|---|---|---|\n| Vista3D | Interactive 3D segmentation | SegResNet-based encoder + SAM-style interactive prompts | Universal (133 structures) | \n| SwinUNETR (BTCV) | Multi-organ CT segmentation | Swin Transformer U-Net | 13 abdominal organs | \n| wholeBody_ct_segmentation | Whole-body CT segmentation | SegResNet | 104 anatomical structures | \n| pancreas_ct_dints_segmentation | Pancreas CT segmentation | DiNTS (differentiable neural architecture search, or NAS) | Pancreas | \n| spleen_deepedit_annotation | Interactive spleen annotation | DynUNet | Spleen | \n\n*Table 1. The five validated bundles.*\n\nEvery bundle runs through the standard `monai.bundle` runner and clears its upstream Dice\nthresholds (the Sørensen–Dice overlap score used to score segmentation accuracy) with no changes to model weights and no changes to inference logic.\n\nPer-bundle compiled-PyTorch latencies range from 3.60 ms (spleen_deepedit) to 3,287 ms (pancreas_dints) on AMD Instinct MI355X (gfx950) with ROCm 10.0, measured on a spleen CT volume (spleen_10.nii.gz). All measurements in this post were taken on AMD Instinct MI355X.\n\n## The Overlay: Optimization Without a Fork[#](#the-overlay-optimization-without-a-fork)\n\nMONAI Model Zoo on ROCm uses MONAI’s native configuration-overlay mechanism. Each validated bundle ships a companion\n`inference_rocm.json` or `inference_rocm.yaml` file alongside the standard `inference.json`. To enable AMD optimizations, pass both configuration files as a JSON list:\n\n```\n# MIOpen: route channels-last convolutions through NHWC kernels and tune\nexport PYTORCH_MIOPEN_SUGGEST_NHWC=1\nexport MIOPEN_FIND_MODE=1\nexport MIOPEN_FIND_ENFORCE=4\n\n# TorchInductor: autotune torch.compile() kernels\nexport TORCHINDUCTOR_MAX_AUTOTUNE=1\nexport TORCHINDUCTOR_MAX_AUTOTUNE_GEMM=1\nexport TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1\nexport TORCHINDUCTOR_EPILOGUE_FUSION=1\nexport TORCHINDUCTOR_MAX_AUTOTUNE_CONV_BACKENDS=ATEN,TRITON\n\n# Bundles load custom code from the bundle's scripts/ package, which must be on\n# the Python path when the configuration is initialized. Run from inside the bundle.\ncd models/BUNDLE_NAME\n\npython -m monai.bundle run \\\n    --config_file \"['configs/inference.json',\n                    'configs/inference_rocm.json']\" \\\n    --bundle_root . \\\n    --dataset_dir /path/to/input_dir \\\n    --output_dir /path/to/output_dir\n```\n\nThe overlay merges into the base configuration at load time, and no upstream files are modified. Configuration files are either in JSON or YAML format, depending on the bundle.\n\nRun the command from inside the bundle directory. MONAI adds the bundle root to the Python path only once `run()` starts, while `initialize()` resolves the bundle’s own `scripts.*` classes before that. A bundle with custom code fails to import when launched from the repository root.\n\nPass `--dataset_dir` and `--output_dir` as absolute paths.\n\n### Three Techniques, Applied Consistently[#](#three-techniques-applied-consistently)\n\nEvery `inference_rocm.*` overlay applies the same three optimizations across all five bundles:\n\n1. Channels-last 3D memory layout ( `memory_format=torch.channels_last_3d` ): aligns tensor memory to the layout the HIP 3D convolution kernels expect, eliminating the redundant transposes that would otherwise wrap every 3D convolution layer.\n2. BF16 automatic mixed precision ( `amp_kwargs={'dtype': torch.bfloat16}` ): uses native\nBF16 on AMD GPUs. Activations and weights move as 16-bit values instead of 32-bit, and BF16 keeps\na wide exponent range.\n3. `torch.compile()` : uses PyTorch’s Inductor backend, which on ROCm triggers hipBLASLt kernel\nselection and graph-level fusion. It is applied at the network level as`torch.compile(@network)` ,\nso the full inference graph is compiled once and cached.\n\nThe Vista3D overlay is representative of the pattern:\n\n``` python\n{\n    \"+imports\": [\"$import torch\"],\n    \"network\": \"$@network_def.to(@device).to(memory_format=torch.channels_last_3d)\",\n    \"initialize\": [\n        \"$monai.utils.set_determinism(seed=123)\",\n        \"$setattr(torch.backends.cudnn, 'deterministic', False)\",\n        \"$setattr(@evaluator, 'amp_kwargs', {'dtype': torch.bfloat16})\",\n        \"$@checkpointloader(@evaluator)\",\n        \"$setattr(@evaluator, 'network', torch.compile(@network))\"\n    ]\n}\n```\n\nThe `+imports` prefix is MONAI bundle syntax for appending to an existing list. Every other key\noverrides its upstream counterpart, and the base `inference.json` stays untouched.\n\nFour of the five bundles need nothing beyond those keys. Only `spleen_deepedit` adds a line to its\nown overlay:\n\n```\n\"$@network_def.enable_gemm_transpose(True) if hasattr(@network_def, 'enable_gemm_transpose') else None\"\n```\n\n## Performance on AMD Instinct MI355X (gfx950)[#](#performance-on-amd-instinct-mi355x-gfx950)\n\nInference latency was measured with the overlay on AMD Instinct, using ordinary PyTorch with no offline engine-compilation step.\n\nTable 3 in the System Configuration section lists the hardware, software, and timing method behind these latency numbers. Table 2 below shows the p50 latency and Dice result for each bundle. All five bundles clear the upstream MONAI Model Zoo Dice thresholds.\n\n| Bundle | MI355X (gfx950) inference latency (ms, p50) | Dice / correctness | \n|---|---|---|\n| Vista3D | 268 | PASS | \n| SwinUNETR (BTCV) | 293 | PASS | \n| wholeBody_ct | 386 | PASS | \n| pancreas_ct_dints | 3,287 | PASS | \n| spleen_deepedit | 3.60 | PASS | \n\n*Table 2. Per-bundle inference latency. p50 is the median: half the timed iterations completed\nfaster than the value shown. Measured over 20 timed iterations after 5 warmup iterations.*\n\n## Memory Capacity for Large Volumes[#](#memory-capacity-for-large-volumes)\n\nSliding-window inference works through a volume in overlapping windows, and the cost is dominated by moving data rather than by arithmetic. That makes available memory a practical constraint: the more of the volume and its activations that stay resident on the GPU, the less the pipeline has to shuttle between host and device between windows. AMD Instinct MI355X carries 288 GB of HBM3E, which is why the largest bundles here, Vista3D at 133 structures and wholeBody_ct at 104, run without the working set spilling.\n\n## Getting Started[#](#getting-started)\n\nThe validated bundles are in the AMD fork of the MONAI Model Zoo in\n[`AMD-Ecosystem/monai-model-zoo`](https://github.com/AMD-Ecosystem/monai-model-zoo).\nYou’ll need:\n\n- An AMD Instinct GPU. The measurements in this post were taken on MI355X; MI300X and MI325X are supported by the release\n- ROCm 10.0\n- Python 3.12\n- MONAI 1.6.0, from the [AMD ROCm package index](https://pypi.amd.com/rocm-10.0.0/simple/) or a[ROCm PyTorch Docker image](https://hub.docker.com/r/rocm/pytorch) . The[MONAI on ROCm installation guide](https://rocm.docs.amd.com/projects/monai/en/docs-26.08/install/installation.html) has the full steps\n\n```\ngit clone https://github.com/AMD-Ecosystem/monai-model-zoo\n\n# MIOpen: route channels-last convolutions through NHWC kernels and tune\nexport PYTORCH_MIOPEN_SUGGEST_NHWC=1\nexport MIOPEN_FIND_MODE=1\nexport MIOPEN_FIND_ENFORCE=4\n\n# TorchInductor: autotune torch.compile() kernels\nexport TORCHINDUCTOR_MAX_AUTOTUNE=1\nexport TORCHINDUCTOR_MAX_AUTOTUNE_GEMM=1\nexport TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1\nexport TORCHINDUCTOR_EPILOGUE_FUSION=1\nexport TORCHINDUCTOR_MAX_AUTOTUNE_CONV_BACKENDS=ATEN,TRITON\n\n# Run from inside the bundle directory so its scripts/ package is importable:\ncd models/BUNDLE_NAME\n\n# Pass both the base configuration and the ROCm overlay as a JSON list:\npython -m monai.bundle run \\\n    --config_file \"['configs/inference.json',\n                    'configs/inference_rocm.json']\" \\\n    --bundle_root . \\\n    --dataset_dir /path/to/input_dir \\\n    --output_dir /path/to/output_dir\n```\n\nCheck the bundle’s own `docs/` directory for its expected input format: the input key varies by\nbundle, and `vista3d` takes `--input_dict` rather than `--dataset_dir`.\n\n## Summary[#](#summary)\n\nIn this blog you explored how five MONAI Model Zoo bundles spanning universal segmentation,\nwhole-body anatomy, pancreas, and spleen run optimized inference on AMD Instinct™ GPUs under\nAMD Life Science 26.08. It showed how the ROCm overlay makes this work without forking a single\nbundle: channels-last 3D, BF16, and `torch.compile()`, applied consistently and layered on top of\neach bundle’s own configuration. All five pass their upstream Dice thresholds, so the speed comes\nwithout an accuracy trade-off.\n\nIn practice, you can run these five existing Model Zoo bundles on an Instinct GPU without a rewrite. Keep the upstream bundle, add the overlay configuration, and measure on your own hardware.\n\nThis is an Early Access release focused on inference, and it is one layer of the AMD Life Science\nmedical-imaging stack. Companion posts cover [the hipCIM I/O library underneath it](https://rocm.blogs.amd.com/software-tools-optimization/hipcim-26-08/README.html) and\n[MONAI data loading](https://rocm.blogs.amd.com/artificial-intelligence/monai-hipcim-2608/README.html), with interactive annotation still to come. Work is continuing on training validation across all\nfive bundles, on additional Model Zoo bundles, and on MIOpen 3D kernel work for the most\nconvolution-heavy architectures, wholeBody_ct and spleen_deepedit, using AMD-optimized MIOpen\nComposable Kernel (CK) 3D kernels. Follow the AMD ROCm Blogs for the rest of the series.\n\n## Additional Resources[#](#additional-resources)\n\n- [hipCIM 26.06: GPU-Accelerated Whole-Slide Image I/O on AMD Instinct™ GPUs](https://rocm.blogs.amd.com/software-tools-optimization/hipcim-26-08/README.html) : the image I/O layer beneath this work.\n- [Accelerating MONAI Digital-Pathology Pipelines on AMD Instinct™ GPUs with hipCIM](https://rocm.blogs.amd.com/artificial-intelligence/monai-hipcim-2608/README.html) : hipCIM as the MONAI`WSIReader` backend.\n- [Project MONAI Model Zoo](https://github.com/Project-MONAI/model-zoo) : the upstream bundle collection, licensed under the Apache License 2.0.\n- [MONAI bundle configuration documentation](https://docs.monai.io/en/stable/config_syntax.html) : the configuration and overlay mechanism used here.\n- [MONAI Model Zoo on GitHub](https://github.com/AMD-Ecosystem/monai-model-zoo)\n- [MONAI on GitHub](https://github.com/AMD-Ecosystem/MONAI)\n- [AMD ROCm documentation](https://rocm.docs.amd.com/)\n- [AMD ROCm Blogs](https://rocm.blogs.amd.com/)\n\n## System Configuration[#](#system-configuration)\n\nAll five bundle latencies were measured on the configuration below, on **21 September 2026**.\n\n| Component | Configuration | \n|---|---|\n| GPU | AMD Instinct™ MI355X (gfx950), bare metal | \n| Host CPU | AMD EPYC™ 9575F, 64 cores | \n| ROCm | 10.0 | \n| PyTorch | 2.13.0+rocm10.0.0 (HIP runtime 7.15.26333) | \n| Triton | 3.8.0+rocm10.0.0 | \n| MONAI | 1.6.0 | \n| NumPy | 2.4.6 | \n| Python | 3.12 | \n| PyTorch path | `torch.compile()` , BF16, channels-last 3D | \n| Inferer | MONAI `SlidingWindowInferer` | \n| Timing | 5 warmup iterations, then 20 timed iterations, reported as p50 | \n| Input | BTCV spleen CT volume, `spleen_10.nii.gz` , for all bundles | \n| Correctness | Dice against the upstream MONAI Model Zoo thresholds | \n\n*Table 3. Hardware, software, and timing method behind the latency numbers.*\n\nThe operating system, amdgpu driver version, and container image digest were not recorded for this run, so they are omitted rather than inferred. Testing conducted by AMD.\n\n## Disclaimers[#](#disclaimers)\n\nPerformance may vary by hardware configuration, workload, and software version.\n\n## Third-Party Content Disclaimer[#](#third-party-content-disclaimer)\n\nThird-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.\n\n- MONAI and the MONAI Model Zoo bundles are © Project MONAI, licensed under the Apache License 2.0.\n- The spleen CT volume `spleen_10.nii.gz` is from the Medical Segmentation Decathlon (Task09 Spleen), licensed CC-BY-SA 4.0.\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/bringing-the-monai-model-zoo-to-amd-instincttm-gpus-validated-inference-on-rocm", "canonical_source": "https://rocm.blogs.amd.com/artificial-intelligence/monai-zoo-2608/README.html", "published_at": "2026-10-30 00:00:00+00:00", "updated_at": "2026-10-30 20:19:58.762532+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "ai-research"], "entities": ["AMD", "MONAI", "AMD Instinct MI355X", "ROCm", "Vista3D", "SwinUNETR", "SegResNet", "PyTorch"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/bringing-the-monai-model-zoo-to-amd-instincttm-gpus-validated-inference-on-rocm", "markdown": "https://wpnews.pro/news/bringing-the-monai-model-zoo-to-amd-instincttm-gpus-validated-inference-on-rocm.md", "text": "https://wpnews.pro/news/bringing-the-monai-model-zoo-to-amd-instincttm-gpus-validated-inference-on-rocm.txt", "jsonld": "https://wpnews.pro/news/bringing-the-monai-model-zoo-to-amd-instincttm-gpus-validated-inference-on-rocm.jsonld"}}