Reproducing AMD MLPerf Training v6.1 Submission Result AMD published a step-by-step guide to reproducing its MLPerf Training v6.1 submission results, covering five benchmarks versus three in v6.0: Llama3.1-8B pretraining, Llama2-70B LoRA fine-tuning, DLRMv4, Flux.1 and GPT-OSS-20B. All AMD first-party results ran on AMD Instinct MI355X and MI350X GPUs, with the language-model and Flux.1 benchmarks powered by Primus, AMD's unified training framework introduced in v6.0, and DLRMv4 using the HSTU stack. The guide specifies convergence targets such as log perplexity ≤ 3.3 for Llama3.1-8B on C4 and validation loss ≤ 0.586 for Flux.1 on CC12M, and requires 8 MI355X GPUs per node, or 4 nodes for Flux.1. Reproducing AMD MLPerf Training v6.1 Submission Result reproducing-amd-mlperf-training-v6-1-submission-result This blog provides a step-by-step guide for reproducing AMD’s MLPerf Training v6.1 submission results. AMD submitted results on five benchmarks this round — an expanded portfolio compared to the three benchmarks submitted in v6.0: - Llama3.1-8B pretraining — full pretraining from random weights on the C4 dataset - Llama2-70B LoRA fine-tuning — parameter-efficient fine-tuning of Llama 2 70B using the GovReport dataset - DLRMv4 — recommendation model training on the Yambda dataset - Flux.1 — text-to-image diffusion model training, with two recipes: a6w4 + fp4 and fp6 + fp4 - GPT-OSS-20B — pretraining of a 20-billion-parameter open-weight large language model All AMD first-party results were submitted on AMD Instinct MI355X and MI350X GPUs. Primus, AMD’s unified training framework introduced in v6.0, powers the language-model and Flux.1 benchmarks in this submission. DLRMv4 uses the HSTU stack. Readers looking for context on the performance results and competitive analysis should refer to the companion blog post: Technical Dive into AMD’s MLPerf Training v6.1 Submission https://rocm.blogs.amd.com/artificial-intelligence/mlperf-training-v6.1/README.html . The table below provides a summary of AMD submissions covered in this blog: | Submission ID | Model | Dataset | Convergence Target | GPUs in This Guide | |---|---|---|---|---| | 6.1-0062 | Llama3.1-8B Pretraining | C4 | Log perplexity ≤ 3.3 | 8 | | 6.1-0062 | Llama2-70B LoRA Fine-Tuning | SCROLLS GovReport | Cross-entropy loss ≤ 0.925 | 8 | | 6.1-0061 | DLRMv4 | Yambda | AUC ≥ 0.75 | 8 | | 6.1-0064 | Flux.1, a6w4 + fp4 recipe | CC12M | Validation loss ≤ 0.586 | 32 | | 6.1-0001 | Flux.1, fp6 + fp4 recipe | CC12M | Validation loss ≤ 0.586 | 32 | | 6.1-0062 | GPT-OSS-20B | C4 | Log perplexity ≤ 3.34 | 8 | Prerequisites prerequisites To follow this guide you need a Linux machine with AMD MI355X GPUs, the Docker image tag for each benchmark see Docker Images below , and this page. For Flux.1, which runs on four nodes under Slurm, check the Slurm login node and every compute node. | Requirement | How to check | If it is missing | |---|---|---| | AMD Instinct MI355X GPUs, 8 per node 4 nodes for Flux.1 | amd-smi list lists 8 GPUs | If fewer than 8 appear, something is wrong with the node. Reboot it or power cycle it first | | Ubuntu 22.04 or later | lsb release -d | Install Ubuntu 22.04 or later | | amdgpu driver versions below | cat /sys/module/amdgpu/version | Install it with the ROCm installation guide https://rocm.docs.amd.com | | ROCm tools on the host | rocm agent enumerator prints the GPU architectures | Install ROCm as above; the launch scripts call this tool | | Idle GPUs | amd-smi monitor shows near-zero GFX use and docker ps lists no containers | Stop the other workloads | | Docker without sudo | docker ps runs | sudo usermod -aG docker $USER , then log out and back in | | Passwordless sudo | sudo -n true && echo OK prints OK | Configure it; the launch scripts use sudo to tune the host | | cpupower | cpupower --version | sudo apt-get install linux-tools-common linux-tools-$ uname -r | | Python 3 with pip | python3 -m pip --version | sudo apt-get install python3-pip | | A writable /data folder | test -w /data && echo OK | sudo mkdir -p /data && sudo chown $USER /data , or use another path in every command | | Internet access | curl -sI https://github.com | Allow Docker Hub, training.mlcommons-storage.org , raw.githubusercontent.com , and github.com | Two benchmarks have extra requirements: - Llama2-70B LoRA: To download the model from HuggingFace, sign the LLAMA 2 COMMUNITY LICENSE AGREEMENT for meta-llama/Llama-2-70b-hf https://huggingface.co/meta-llama/Llama-2-70b-hf and create a HuggingFace token in your token settings https://huggingface.co/settings/tokens . Steps 2 and 4 use the token as HF TOKEN . - Flux.1: Slurm, shared storage that every node can reach, and an RDMA network between the nodes ls /dev/infiniband lists the RDMA devices on each node . The launcher stops if the container cannot open an RDMA device, because the job would otherwise run over TCP, too slowly to finish. AMD’s submitted runs used the ROCm versions specified in the table below. The ROCm version is the one inside each Docker image, so the ROCm installed on your host does not need to match it. | Model | amdgpu driver host | ROCm in the image | |---|---|---| | Llama3.1-8B | 6.19.4 | 7.15.0 | | Llama2-70B LoRA | 6.19.4 | 7.15.0 | | DLRMv4 | 6.19.4 | 7.2.1 | | Flux.1, a6w4 + fp4 recipe | 6.16.6 | 7.15.0 | | Flux.1, fp6 + fp4 recipe | 6.14.14 | 10.0.0 | | GPT-OSS-20B | 6.19.4 | 7.15.0 | Note Before every run, the launch scripts run runtime tunables.sh on the host with sudo . It drops the filesystem caches, loads the amdgpu driver, disables CPU idle state 2, sets the CPU governor to performance , disables the NMI watchdog, NUMA balancing, and address-space randomization, and enables transparent huge pages. These settings last until the machine reboots, so use a machine that you are allowed to tune. Disk Space, Time, and Where Things Live disk-space-time-and-where-things-live Before downloading anything, check free space with df -h on /data and on the Docker root folder docker info --format '{{.DockerRootDir}}' to ensure you have enough space to run the benchmark. | Model | Image | Data and model | Total | |---|---|---|---| | Llama3.1-8B | 46 GB | 117 GB 85 GB dataset + 32 GB model | about 163 GB | | Llama2-70B LoRA | 47 GB | about 270 GB | about 320 GB | | DLRMv4 | 86 GB | 245 GB 227 GB dataset + 18 GB index | about 330 GB | | GPT-OSS-20B | 43 GB | 117 GB 85 GB dataset + 32 GB model | about 160 GB | | Flux.1, a6w4 + fp4 recipe | 45 GB per node | about 2.5 TB shared, plus about 2.5 TB of local disk per node | about 2.5 TB shared, plus about 2.55 TB on each node | | Flux.1, fp6 + fp4 recipe | 63 GB per node | about 2.4 TB shared | about 2.4 TB shared, plus about 63 GB on each node | Llama3.1-8B and GPT-OSS-20B use the same dataset and tokenizer, so one download can serve both by pointing DATADIR and MODELDIR at the same location for both models. Trial times from AMD’s submitted runs are included below for reference. A trial takes longer than its time to train because startup model construction, data setup, and warm-up happens before the timed region. | Model | One trial, launch to finish | Of which startup | A full 10-trial set | |---|---|---|---| | Llama3.1-8B | about 77 min | about 7 min | about 13 hours | | Llama2-70B LoRA | about 18 min | about 11 min | about 3 hours | | DLRMv4 | about 25 min | about 4 min | about 4 hours | | GPT-OSS-20B | about 61 min | about 7 min | about 10 hours | | Flux.1, a6w4 + fp4 recipe | about 54 min | about 6 min | about 9.5 hours | | Flux.1, fp6 + fp4 recipe | about 52 min | about 4 min | about 9 hours | Get the Implementation Code get-the-implementation-code The benchmark code and launch scripts are inside each Docker image, in /workspace/code . Step 1 of each benchmark copies that folder to the host ~/mlperf/