{"slug": "reproducing-amd-mlperf-training-v6-1-submission-result", "title": "Reproducing AMD MLPerf Training v6.1 Submission Result", "summary": "AMD published a step-by-step guide to reproducing its MLPerf Training v6.1 submission results, covering five benchmarks versus three in v6.0: Llama3.1-8B pretraining, Llama2-70B LoRA fine-tuning, DLRMv4, Flux.1 and GPT-OSS-20B. All AMD first-party results ran on AMD Instinct MI355X and MI350X GPUs, with the language-model and Flux.1 benchmarks powered by Primus, AMD's unified training framework introduced in v6.0, and DLRMv4 using the HSTU stack. The guide specifies convergence targets such as log perplexity ≤ 3.3 for Llama3.1-8B on C4 and validation loss ≤ 0.586 for Flux.1 on CC12M, and requires 8 MI355X GPUs per node, or 4 nodes for Flux.1.", "body_md": "# Reproducing AMD MLPerf Training v6.1 Submission Result[#](#reproducing-amd-mlperf-training-v6-1-submission-result)\n\nThis blog provides a step-by-step guide for reproducing AMD’s MLPerf Training v6.1 submission results. AMD submitted results on five benchmarks this round — an expanded portfolio compared to the three benchmarks submitted in v6.0:\n\n- **Llama3.1-8B pretraining** — full pretraining from random weights on the C4 dataset\n- **Llama2-70B LoRA fine-tuning** — parameter-efficient fine-tuning of Llama 2 70B using the GovReport dataset\n- **DLRMv4** — recommendation model training on the Yambda dataset\n- **Flux.1** — text-to-image diffusion model training, with two recipes: a6w4 + fp4 and fp6 + fp4\n- **GPT-OSS-20B** — pretraining of a 20-billion-parameter open-weight large language model\n\nAll AMD first-party results were submitted on AMD Instinct **MI355X** and **MI350X** GPUs. Primus, AMD’s unified training framework introduced in v6.0, powers the language-model and Flux.1 benchmarks in this submission. DLRMv4 uses the HSTU stack.\nReaders looking for context on the performance results and competitive analysis should refer to the companion blog post: [Technical Dive into AMD’s MLPerf Training v6.1 Submission](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-training-v6.1/README.html).\n\nThe table below provides a summary of AMD submissions covered in this blog:\n\n| Submission ID | Model | Dataset | Convergence Target | GPUs in This Guide | \n|---|---|---|---|---|\n| 6.1-0062 | Llama3.1-8B Pretraining | C4 | Log perplexity ≤ 3.3 | 8 | \n| 6.1-0062 | Llama2-70B LoRA Fine-Tuning | SCROLLS GovReport | Cross-entropy loss ≤ 0.925 | 8 | \n| 6.1-0061 | DLRMv4 | Yambda | AUC ≥ 0.75 | 8 | \n| 6.1-0064 | Flux.1, a6w4 + fp4 recipe | CC12M | Validation loss ≤ 0.586 | 32 | \n| 6.1-0001 | Flux.1, fp6 + fp4 recipe | CC12M | Validation loss ≤ 0.586 | 32 | \n| 6.1-0062 | GPT-OSS-20B | C4 | Log perplexity ≤ 3.34 | 8 | \n\n## Prerequisites[#](#prerequisites)\n\nTo follow this guide you need a Linux machine with AMD MI355X GPUs, the Docker image tag for each benchmark (see **Docker Images** below), and this page. For Flux.1, which runs on four nodes under Slurm, check the Slurm login node and every compute node.\n\n| Requirement | How to check | If it is missing | \n|---|---|---|\n| AMD Instinct MI355X GPUs, 8 per node (4 nodes for Flux.1) | `amd-smi list` lists 8 GPUs | If fewer than 8 appear, something is wrong with the node. Reboot it or power cycle it first | \n| Ubuntu 22.04 or later | `lsb_release -d` | Install Ubuntu 22.04 or later | \n| amdgpu driver (versions below) | `cat /sys/module/amdgpu/version` | Install it with the [ROCm installation guide](https://rocm.docs.amd.com) | \n| ROCm tools on the host | `rocm_agent_enumerator` prints the GPU architectures | Install ROCm as above; the launch scripts call this tool | \n| Idle GPUs | `amd-smi monitor` shows near-zero GFX use and`docker ps` lists no containers | Stop the other workloads | \n| Docker without `sudo` | `docker ps` runs | `sudo usermod -aG docker $USER` , then log out and back in | \n| Passwordless `sudo` | `sudo -n true && echo OK` prints`OK` | Configure it; the launch scripts use `sudo` to tune the host | \n| `cpupower` | `cpupower --version` | `sudo apt-get install linux-tools-common linux-tools-$(uname -r)` | \n| Python 3 with pip | `python3 -m pip --version` | `sudo apt-get install python3-pip` | \n| A writable `/data` folder | `test -w /data && echo OK` | `sudo mkdir -p /data && sudo chown $USER /data` , or use another path in every command | \n| Internet access | `curl -sI https://github.com` | Allow Docker Hub, `training.mlcommons-storage.org` ,`raw.githubusercontent.com` , and`github.com` | \n\nTwo benchmarks have extra requirements:\n\n- **Llama2-70B LoRA:** To download the model from HuggingFace, sign the LLAMA 2 COMMUNITY LICENSE AGREEMENT for[`meta-llama/Llama-2-70b-hf`](https://huggingface.co/meta-llama/Llama-2-70b-hf) and create a HuggingFace token in your[token settings](https://huggingface.co/settings/tokens) . Steps 2 and 4 use the token as`HF_TOKEN` .\n- **Flux.1:** Slurm, shared storage that every node can reach, and an RDMA network between the nodes (`ls /dev/infiniband` lists the RDMA devices on each node). The launcher stops if the container cannot open an RDMA device, because the job would otherwise run over TCP, too slowly to finish.\n\nAMD’s submitted runs used the ROCm versions specified in the table below. The ROCm version is the one inside each Docker image, so the ROCm installed on your host does not need to match it.\n\n| Model | amdgpu driver (host) | ROCm (in the image) | \n|---|---|---|\n| Llama3.1-8B | 6.19.4 | 7.15.0 | \n| Llama2-70B LoRA | 6.19.4 | 7.15.0 | \n| DLRMv4 | 6.19.4 | 7.2.1 | \n| Flux.1, a6w4 + fp4 recipe | 6.16.6 | 7.15.0 | \n| Flux.1, fp6 + fp4 recipe | 6.14.14 | 10.0.0 | \n| GPT-OSS-20B | 6.19.4 | 7.15.0 | \n\nNote\n\nBefore every run, the launch scripts run `runtime_tunables.sh` on the host with `sudo`. It drops the filesystem caches, loads the `amdgpu` driver, disables CPU idle state 2, sets the CPU governor to `performance`, disables the NMI watchdog, NUMA balancing, and address-space randomization, and enables transparent huge pages. These settings last until the machine reboots, so use a machine that you are allowed to tune.\n\n### Disk Space, Time, and Where Things Live[#](#disk-space-time-and-where-things-live)\n\nBefore downloading anything, check free space with `df -h` on `/data` and on the Docker root folder (`docker info --format '{{.DockerRootDir}}'`) to ensure you have enough space to run the benchmark.\n\n| Model | Image | Data and model | Total | \n|---|---|---|---|\n| Llama3.1-8B | 46 GB | 117 GB (85 GB dataset + 32 GB model) | about 163 GB | \n| Llama2-70B LoRA | 47 GB | about 270 GB | about 320 GB | \n| DLRMv4 | 86 GB | 245 GB (227 GB dataset + 18 GB index) | about 330 GB | \n| GPT-OSS-20B | 43 GB | 117 GB (85 GB dataset + 32 GB model) | about 160 GB | \n| Flux.1, a6w4 + fp4 recipe | 45 GB per node | about 2.5 TB shared, plus about 2.5 TB of local disk per node | about 2.5 TB shared, plus about 2.55 TB on each node | \n| Flux.1, fp6 + fp4 recipe | 63 GB per node | about 2.4 TB shared | about 2.4 TB shared, plus about 63 GB on each node | \n\nLlama3.1-8B and GPT-OSS-20B use the same dataset and tokenizer, so one download can serve both by pointing `DATADIR` and `MODELDIR` at the same location for both models.\n\nTrial times from AMD’s submitted runs are included below for reference. A trial takes longer than its time to train because startup (model construction, data setup, and warm-up) happens before the timed region.\n\n| Model | One trial, launch to finish | Of which startup | A full 10-trial set | \n|---|---|---|---|\n| Llama3.1-8B | about 77 min | about 7 min | about 13 hours | \n| Llama2-70B LoRA | about 18 min | about 11 min | about 3 hours | \n| DLRMv4 | about 25 min | about 4 min | about 4 hours | \n| GPT-OSS-20B | about 61 min | about 7 min | about 10 hours | \n| Flux.1, a6w4 + fp4 recipe | about 54 min | about 6 min | about 9.5 hours | \n| Flux.1, fp6 + fp4 recipe | about 52 min | about 4 min | about 9 hours | \n\n### Get the Implementation Code[#](#get-the-implementation-code)\n\nThe benchmark code and launch scripts are inside each Docker image, in `/workspace/code`. Step 1 of each benchmark copies that folder to the host (`~/mlperf/<benchmark>_code`, or `/shared/mlperf_flux1/flux1_code` for the Flux.1 a6w4 + fp4 recipe and `/shared/mlperf_flux1_fp6/flux1_fp6_code` for the Flux.1 fp6 + fp4 recipe), and the launch scripts run that host copy.\n\n### Docker Images[#](#docker-images)\n\nEach benchmark has its own docker image that is available on [docker hub](https://hub.docker.com/r/rocm/amd-mlperf/tags):\n\n| Model | Image | Image size | \n|---|---|---|\n| Llama2-70B LoRA | `rocm/amd-mlperf:llama2_70b_training_6.1` | 47.4 GB | \n| Llama3.1-8B | `rocm/amd-mlperf:llama31_8b_training_6.1` | 46.2 GB | \n| DLRMv4 | `rocm/amd-mlperf:dlrmv4_training_6.1` | 85.7 GB | \n| Flux.1, a6w4 + fp4 recipe | `rocm/amd-mlperf:mlperf_flux1_6.1` | 45.6 GB | \n| Flux.1, fp6 + fp4 recipe | Built with Dockerfile (`mlperf_flux1_fp6:6.1` ) | 63 GB | \n| GPT-OSS-20B | `rocm/amd-mlperf:gpt_oss_20b_training_6.1` | 42.9 GB | \n\nIf you prefer to build the docker images yourself, please follow the instruction in the [Appendix](#appendix-build-the-docker-images-instead-of-pulling).\n\n## How Results Are Checked[#](#how-results-are-checked)\n\nAll five benchmarks are scored the same way. This section explains the method once, and the final step of each benchmark shows how to apply it to that benchmark’s logs.\n\n### What the Logs Record[#](#what-the-logs-record)\n\nEach trial writes one log, `$LOGDIR/<DATESTAMP>_<N>.log`, where `<DATESTAMP>` is the launch time as a string of digits and `<N>` is the trial number, which starts at 1 (Flux.1 starts at 0). The commands in each benchmark’s Step 5 use a `*` in place of the timestamp, so `\"$LOGDIR\"/*_1.log` selects trial 1 (`*_0.log` for Flux.1); change the number to read another trial. They also use `$LOGDIR` from Step 3, so set it again if you opened a new terminal. Scoring uses only the MLPerf log lines, which start with `:::MLLOG` followed by a JSON record. The following three keys in the record are important to determine the result:\n\n- `run_start` marks the start of the timed region, after initialization (model construction, data setup, and warm-up) has finished.\n- `eval_accuracy` holds the quality metric from one evaluation. Its`metadata.samples_count` is the number of training samples consumed up to that evaluation.\n- `run_stop` marks the end of the timed region. Its`metadata.status` is`success` when an evaluation met the quality target, and`aborted` when training reached its step limit first.\n\nFor example, these are the last two records of a submitted Llama3.1-8B trial, which reached the target at its 15th evaluation:\n\n```\n:::MLLOG {\"namespace\": \"\", \"time_ms\": 1792039530706, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_accuracy\", \"value\": 3.299457550048828, \"metadata\": {\"file\": \"/opt/venv/lib/python3.12/site-packages/primus_mllog/mlperf_pretrain_trainer.py\", \"lineno\": 336, \"samples_count\": 184320}}\n:::MLLOG {\"namespace\": \"\", \"time_ms\": 1792039530840, \"event_type\": \"INTERVAL_END\", \"key\": \"run_stop\", \"value\": null, \"metadata\": {\"file\": \"/opt/venv/lib/python3.12/site-packages/primus_mllog/mlperf_pretrain_trainer.py\", \"lineno\": 216, \"samples_count\": 184320, \"status\": \"success\"}}\n```\n\n### Quality, Time to Train, and Score[#](#quality-time-to-train-and-score)\n\nTo understand how MLPerf Training submissions are evaluated, it is important to understand the following metrics:\n\n- **Quality.** Training pauses at a fixed interval of training samples and evaluates the model on a fixed validation set. Each benchmark’s final step gives its metric, interval, and validation set. Training stops at the first evaluation that meets the target and logs`run_stop` with`status: success` .\n- **Time to train.** A trial’s time to train is the`run_stop` timestamp minus the`run_start` timestamp. It includes the time spent in evaluations, but not initialization.\n- **Score.** A submission is a set of 10 trials. The fastest and slowest trials are dropped, and the score is the mean time to train of the remaining eight. All 10 trials should converge.\n\n### Convergence and Log Checks[#](#convergence-and-log-checks)\n\nTo simplify the scoring process, MLCommons provides two checkers, in the `mlperf_logging` package:\n\n- **RCP checker.** Reference convergence points (RCPs) record how many samples the reference implementation needs to converge at a given global batch size. The RCP checker compares the mean samples-to-converge of your 10-trial set against them and ends with`RCP test PASSED` or`RCP test FAILED` . If a set converges in fewer samples than the reference mean but is still within the allowed range, it passes, and the checker reports a normalization factor that scales the score up.\n- **Compliance checker.** This checks that a trial log contains every required key with allowed values, for example the hyperparameters fixed by the rules. The Llama3.1-8B, DLRMv4, Flux.1, and GPT-OSS-20B launchers run it after every trial and append the result to`$LOGDIR/compliance_<DATESTAMP>.out` , where each trial’s section ends with`INFO - SUCCESS` or`ERROR - FAILED` . The Llama2-70B LoRA launcher skips it, so that benchmark’s final step shows how to run it on the host.\n\nInstall the checkers on the host once:\n\n```\npython3 -m pip install \"git+https://github.com/mlcommons/logging.git@6.1.0-rc2\"\n```\n\nThe RCP checker reads only files named `result_<N>.txt`, so each benchmark’s final step copies the trial logs to those names before running it. It needs a full set of 10 trials: with fewer logs it prints `nan` and `RCP test FAILED`, so skip it after a single test trial. Use a fresh `$LOGDIR` for each 10-trial set, so that the glob `\"$LOGDIR\"/*_[0-9]*.log` matches only that set.\n\n## Benchmark 1: Llama3.1-8B Pretraining[#](#benchmark-1-llama3-1-8b-pretraining)\n\n### Background - Llama3.1-8B[#](#background-llama3-1-8b)\n\nThe Llama3.1-8B pretraining benchmark measures how quickly a model can be trained from **random weights** to reach a target validation loss on the C4 (Colossal Cleaned Common Crawl) dataset. AMD led the development of this benchmark in MLPerf Training. No checkpoint conversion is required. **Disk space required:** ~85 GB dataset + ~32 GB model directory.\n\n### Step 1: Pull the Docker Image and Copy the Code - Llama3.1-8B[#](#step-1-pull-the-docker-image-and-copy-the-code-llama3-1-8b)\n\n```\nmkdir -p ~/mlperf && cd ~/mlperf\n\ndocker pull rocm/amd-mlperf:llama31_8b_training_6.1\n\ncontainer_id=$(docker create rocm/amd-mlperf:llama31_8b_training_6.1) && \\\ndocker cp $container_id:/workspace/code ./llama31_8b_code && \\\ndocker rm $container_id\n```\n\nTo build the image instead, see the **Build the Images Instead of Pulling** section at the end of this guide.\n\n### Step 2: Prepare Dataset and Tokenizer - Llama3.1-8B[#](#step-2-prepare-dataset-and-tokenizer-llama3-1-8b)\n\n```\nmkdir -p /data/mlperf_llama31_8b\ncd /data/mlperf_llama31_8b\n\n# Download training data\nbash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \\\n    -d data https://training.mlcommons-storage.org/metadata/llama-3-1-8b-preprocessed-c4-dataset.uri\n\n# Download model tokenizer\nbash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \\\n    -d model https://training.mlcommons-storage.org/metadata/llama-3-1-8b-tokenizer.uri\n```\n\nAfter the download is complete, the data directory contains files ending in both `.idx` and `.bin` with the following naming conventions:\n\n- Training partitions: `c4-train.en_6_text_document`\n- Validation partitions: `c4-validation-91205-samples.en_text_document`\n\n### Step 3: Set Up Environment - Llama3.1-8B[#](#step-3-set-up-environment-llama3-1-8b)\n\nEnter the `llama31_8b_code` folder that you copied in Step 1, then set the paths and the image:\n\n```\ncd ~/mlperf/llama31_8b_code\n\nmkdir -p /data/mlperf_llama31_8b/results\n\nexport DATADIR=/data/mlperf_llama31_8b/data\nexport MODELDIR=/data/mlperf_llama31_8b/model/\nexport LOGDIR=/data/mlperf_llama31_8b/results\nexport CONT=rocm/amd-mlperf:llama31_8b_training_6.1\n\nsudo chmod -R 777 $LOGDIR\n```\n\nSource the MI355X configuration file:\n\n```\nsource config_MI355X_1x8x1.sh\n```\n\nThe `source` command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.\n\n### Step 4: Run Training - Llama3.1-8B[#](#step-4-run-training-llama3-1-8b)\n\nRun one trial first to confirm that everything works. It takes about 77 minutes, including about 7 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows `ENDING TIMING RUN` followed by `RESULT,LLAMA3.1_8B,,<seconds>,AMD,<start time>`, the total run time in seconds including startup.\n\n```\nexport NEXP=1\nbash run_with_docker.sh\n```\n\nFor a compliant 10-run submission (about 13 hours):\n\n```\nexport NEXP=10\nbash run_with_docker.sh\n```\n\nAfter completion, the trial logs are in `$LOGDIR`.\n\n### Step 5: Check Quality and Validate Results - Llama3.1-8B[#](#step-5-check-quality-and-validate-results-llama3-1-8b)\n\nQuality is validation log perplexity on held-out C4 text (lower is better), evaluated every 12,288 training sequences. A trial succeeds at the first evaluation at or below **3.3**.\n\nPrint the trial’s start, evaluation, and stop records:\n\n```\ngrep -E '\"key\": \"(run_start|eval_accuracy|run_stop)\"' \"$LOGDIR\"/*_1.log\n```\n\nThe trial converged if the last `eval_accuracy` value is 3.3 or lower and `run_stop` shows `\"status\": \"success\"`. Its time to train is the `run_stop` `time_ms` minus the `run_start` `time_ms`, in milliseconds. Across the 10 submitted trials, the target was met at the 15th or 16th evaluation (184,320–196,608 samples), with times to train of 69.9–74.8 minutes.\n\nThe last throughput record gives the average training throughput; expect approximately **44 samples/second** on MI355X:\n\n```\ngrep overall_throughput \"$LOGDIR\"/*_1.log | tail -1\n```\n\nTo score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored **72.94 minutes**. Then run the RCP checker, after copying each trial log to the `result_<N>.txt` name it reads:\n\n```\ni=0; for f in \"$LOGDIR\"/*_[0-9]*.log; do cp \"$f\" \"$LOGDIR/result_$((i++)).txt\"; done\npython3 -m mlperf_logging.rcp_checker \"$LOGDIR\" --rcp_version 6.1.0\n```\n\nFinally, confirm that every trial’s section of `$LOGDIR/compliance_*.out` ends with `INFO - SUCCESS`.\n\n## Benchmark 2: Llama2-70B LoRA Fine-Tuning[#](#benchmark-2-llama2-70b-lora-fine-tuning)\n\n### Background - Llama2-70B LoRA[#](#background-llama2-70b-lora)\n\nThe Llama2-70B LoRA fine-tuning benchmark measures how quickly the Llama 2 70B model can be fine-tuned to reach the target cross-entropy loss on the SCROLLS GovReport summarization dataset (pre-tokenized, packed to sequence length 8,192), using Low-Rank Adaptation (LoRA). **Disk space required:** ~270 GB for model download and conversion.\n\nDownloading the model needs a HuggingFace token (`HF_TOKEN`) with access to `meta-llama/Llama-2-70b-hf`; see **Prerequisites**.\n\n### Step 1: Pull the Docker Image and Copy the Code - Llama2-70B LoRA[#](#step-1-pull-the-docker-image-and-copy-the-code-llama2-70b-lora)\n\n```\nmkdir -p ~/mlperf && cd ~/mlperf\n\ndocker pull rocm/amd-mlperf:llama2_70b_training_6.1\n\ncontainer_id=$(docker create rocm/amd-mlperf:llama2_70b_training_6.1) && \\\ndocker cp $container_id:/workspace/code ./llama2_70b_code && \\\ndocker rm $container_id\n```\n\nTo build the image instead, see the **Build the Images Instead of Pulling** section at the end of this guide.\n\n### Step 2: Download and Preprocess Data - Llama2-70B LoRA[#](#step-2-download-and-preprocess-data-llama2-70b-lora)\n\nStart a container with the host download directory mounted at `/data`. The preparation script is the copy in the image. The commands below are run in the shell you start in, so the folder you are in does not matter:\n\n```\nmkdir -p /data/mlperf_llama2/data\n\ndocker run --rm -it -v /data/mlperf_llama2/data:/data \\\n    --net=host --uts=host \\\n    --ipc=host --device /dev/dri --device /dev/kfd \\\n    --security-opt=seccomp=unconfined \\\n    rocm/amd-mlperf:llama2_70b_training_6.1\n```\n\nFrom inside the container, run the preparation script `prepare_data_and_model.sh`:\n\n```\nexport HF_TOKEN=<your_huggingface_token>\nbash ./scripts/prepare_data_and_model.sh\n```\n\nExit the container. The data directory on the host should look like this:\n\n```\n/data/mlperf_llama2/data\n├── train.npy                          # Packed training data (seq_length=8192)\n├── validation.npy                     # Packed validation data\n├── packed_metadata.jsonl              # Sequence metadata for Megatron-Bridge\n└── megatron_checkpoints/\n    └── Llama-2-70b-hf/\n        ├── latest_checkpointed_iteration.txt\n        ├── latest_train_state.pt\n        └── iter_0000000/\n            ├── __0_0.distcp           # Sharded weights (~64 GB)\n            ├── __0_1.distcp           # Sharded weights (~64 GB)\n            ├── common.pt\n            ├── .metadata\n            ├── metadata.json\n            ├── run_config.yaml\n            ├── modelopt_run_config.yaml\n            ├── train_state.pt\n            └── tokenizer/\n                ├── tokenizer.model\n                ├── tokenizer_config.json\n                └── special_tokens_map.json\n```\n\n### Step 3: Set Up Environment - Llama2-70B LoRA[#](#step-3-set-up-environment-llama2-70b-lora)\n\nEnter the `llama2_70b_code` folder that you copied in Step 1, then set the paths and the image:\n\n```\ncd ~/mlperf/llama2_70b_code\n\nmkdir -p /data/mlperf_llama2/results\n\nexport DATADIR=/data/mlperf_llama2\nexport MODELDIR=/data/mlperf_llama2/data\nexport LOGDIR=/data/mlperf_llama2/results\nexport CONT=rocm/amd-mlperf:llama2_70b_training_6.1\n\nsudo chmod -R 777 $LOGDIR\n```\n\nSource the MI355X configuration:\n\n```\nsource config_MI355X_1x8x1.sh\n```\n\nThe `source` command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.\n\n### Step 4: Run Training - Llama2-70B LoRA[#](#step-4-run-training-llama2-70b-lora)\n\nRun one trial first to confirm that everything works. It takes about 18 minutes, including about 11 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows `ENDING TIMING RUN` followed by `RESULT,LLAMA2_70B_LORA_SFT,,<seconds>,AMD,<start time>`, the total run time in seconds including startup.\n\n```\nexport HF_TOKEN=<your_huggingface_token>\nexport NEXP=1\nbash run_with_docker.sh\n```\n\nFor a compliant 10-run submission (about 3 hours):\n\n```\nexport HF_TOKEN=<your_huggingface_token>\nexport NEXP=10\nbash run_with_docker.sh\n```\n\nAfter completion, the trial logs are in `$LOGDIR`.\n\n### Step 5: Check Quality and Validate Results - Llama2-70B LoRA[#](#step-5-check-quality-and-validate-results-llama2-70b-lora)\n\nQuality is the cross-entropy loss on the GovReport validation set (lower is better), evaluated every 384 training samples after the first three evaluations are skipped. A trial succeeds at the first evaluation at or below **0.925**.\n\nPrint the trial’s start, evaluation, and stop records:\n\n```\ngrep -E '\"key\": \"(run_start|eval_accuracy|run_stop)\"' \"$LOGDIR\"/*_1.log\n```\n\nThe trial converged if the last `eval_accuracy` value is 0.925 or lower and `run_stop` shows `\"status\": \"success\"`. Its time to train is the `run_stop` `time_ms` minus the `run_start` `time_ms`, in milliseconds. All 10 submitted trials met the target at the fifth evaluation (3,072 samples), with a final loss of 0.907–0.913 and times to train of 6.84–6.86 minutes.\n\nThe last throughput record gives the average training throughput; expect approximately **7.5 samples/second** on MI355X:\n\n```\ngrep '\"key\": \"throughput\"' \"$LOGDIR\"/*_1.log | tail -1\n```\n\nTo score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored **6.98 minutes**. Then run the RCP checker, after copying each trial log to the `result_<N>.txt` name it reads:\n\n```\ni=0; for f in \"$LOGDIR\"/*_[0-9]*.log; do cp \"$f\" \"$LOGDIR/result_$((i++)).txt\"; done\npython3 -m mlperf_logging.rcp_checker \"$LOGDIR\" --rcp_version 6.1.0\n```\n\nThe launcher does not run the compliance checker, so run it on the host for each trial log. Every trial should print `INFO - SUCCESS`:\n\n```\nfor f in \"$LOGDIR\"/*_[0-9]*.log; do\n    python3 -m mlperf_logging.compliance_checker --usage training --ruleset 6.1.0 \"$f\" 2>&1 | grep -E \"SUCCESS|FAILED\"\ndone\n```\n\n## Benchmark 3: DLRMv4[#](#benchmark-3-dlrmv4)\n\n### Background - DLRMv4[#](#background-dlrmv4)\n\nDLRMv4 is the latest generation of the Deep Learning Recommendation Model benchmark in MLPerf Training. It trains an HSTU ranking model on the Yambda-5b music-recommendation dataset and is one of the computationally heaviest benchmarks in the suite due to its large embedding tables and high memory requirements. **Disk space required:** ~227 GB for the prepared dataset, plus an ~18 GB time-window index built before the first run.\n\n**Note:** DLRMv4 in MLPerf Training v6.1 uses the **Yambda** dataset — a change from the Criteo Terabyte dataset used by prior DLRM benchmarks. The prepared dataset is published by MLCommons at [`dlrmv4_dataset.uri`](https://training.mlcommons-storage.org/metadata/dlrmv4_dataset.uri).\n\n### Step 1: Pull the Docker Image and Copy the Code - DLRMv4[#](#step-1-pull-the-docker-image-and-copy-the-code-dlrmv4)\n\n```\nmkdir -p ~/mlperf && cd ~/mlperf\n\ndocker pull rocm/amd-mlperf:dlrmv4_training_6.1\n\ncontainer_id=$(docker create rocm/amd-mlperf:dlrmv4_training_6.1) && \\\ndocker cp $container_id:/workspace/code ./dlrmv4_code && \\\ndocker rm $container_id\n```\n\nTo build the image instead, see the **Build the Images Instead of Pulling** section at the end of this guide.\n\n### Step 2: Prepare Dataset - DLRMv4[#](#step-2-prepare-dataset-dlrmv4)\n\n```\nmkdir -p /data/mlperf_dlrmv4\ncd /data/mlperf_dlrmv4\n\nbash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \\\n    -d data https://training.mlcommons-storage.org/metadata/dlrmv4_dataset.uri\n```\n\nAfter the download is complete, the data directory should contain the following folders:\n\n- Training cache: `processed_5b/hstu_cache_L4086/`\n- Item popularity: `processed_5b/item_popularity.npy`\n- Item mappings: `shared_metadata/album_item_mapping.parquet` and`shared_metadata/artist_item_mapping.parquet`\n\nVerify the checksums and build the time-window index once before the first run. The first command runs from the `dlrmv4_code` folder that you copied in Step 1:\n\n```\ncd ~/mlperf/dlrmv4_code\n\nDLRM_DATA_PATH=/data/mlperf_dlrmv4/data bash scripts/verify_dataset.sh\n\ndocker run --rm -v /data/mlperf_dlrmv4/data:/data \\\n    rocm/amd-mlperf:dlrmv4_training_6.1 \\\n    python3 -m generative_recommenders.dlrm_v4.datasets.offline_ts_indices \\\n    --cache-dir /data/processed_5b/hstu_cache_L4086 \\\n    --history-length 4086 --min-history 4086 --window-seconds 86400\n```\n\n### Step 3: Set Up Environment - DLRMv4[#](#step-3-set-up-environment-dlrmv4)\n\nEnter the `dlrmv4_code` folder that you copied in Step 1, then set the paths and the image:\n\n```\ncd ~/mlperf/dlrmv4_code\n\nmkdir -p /data/mlperf_dlrmv4/results\n\nexport DATADIR=/data/mlperf_dlrmv4/data\nexport LOGDIR=/data/mlperf_dlrmv4/results\nexport CONT=rocm/amd-mlperf:dlrmv4_training_6.1\n\nsudo chmod -R 777 $LOGDIR\n```\n\nSource the MI355X configuration:\n\n```\nsource config_MI355X_1x8x1.sh\n```\n\nThe `source` command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.\n\n### Step 4: Run Training - DLRMv4[#](#step-4-run-training-dlrmv4)\n\nRun one trial first to confirm that everything works. It takes about 25 minutes, including about 4 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows `ENDING TIMING RUN` followed by `RESULT,DLRMV4,,<seconds>,AMD,<start time>`, the total run time in seconds including startup.\n\n```\nexport NEXP=1\nbash run_with_docker.sh\n```\n\nFor a compliant 10-run submission (about 4 hours):\n\n```\nexport NEXP=10\nbash run_with_docker.sh\n```\n\nTo run the same single-node configuration through Slurm instead, edit `job.sh` to set `--partition`, `CONT`, `DATADIR`, and `LOGDIR`, then submit it. `job.sh` is set for 1 node and `DGXSYSTEM=MI355X_1x8x1`, and runs 10 trials by default (` NEXP=10`). `CONT` must be an image every node can pull. Run the following Slurm command to submit the job:\n\n```\nsbatch job.sh\n```\n\n### Step 5: Check Quality and Validate Results - DLRMv4[#](#step-5-check-quality-and-validate-results-dlrmv4)\n\nQuality is the AUC of the model’s `listen_plus` prediction on a held-out time window (higher is better), evaluated every 2,293,760 training samples after the first 57,344,000. A trial succeeds at the first evaluation at or above **0.75**.\n\nPrint the trial’s start, evaluation, and stop records:\n\n```\ngrep -E '\"key\": \"(run_start|eval_accuracy|run_stop)\"' \"$LOGDIR\"/*_1.log\n```\n\nThe trial converged if the last `eval_accuracy` value is 0.75 or higher and `run_stop` shows `\"status\": \"success\"`. Its time to train is the `run_stop` `time_ms` minus the `run_start` `time_ms`, in milliseconds. Convergence varies more between trials than in the other benchmarks: across the 10 submitted trials, the target was met after 4 to 15 evaluations (64,225,280–89,456,640 samples), with times to train of 16.5–25.5 minutes.\n\nThe last throughput record gives the average training throughput; expect approximately **62,400 samples/second** on MI355X:\n\n```\ngrep overall_throughput \"$LOGDIR\"/*_1.log | tail -1\n```\n\nTo score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored **18.95 minutes**. Then run the RCP checker, after copying each trial log to the `result_<N>.txt` name it reads:\n\n```\ni=0; for f in \"$LOGDIR\"/*_[0-9]*.log; do cp \"$f\" \"$LOGDIR/result_$((i++)).txt\"; done\npython3 -m mlperf_logging.rcp_checker \"$LOGDIR\" --rcp_version 6.1.0\n```\n\nFinally, confirm that every trial’s section of `$LOGDIR/compliance_*.out` ends with `INFO - SUCCESS`.\n\n## Benchmark 4: Flux.1[#](#benchmark-4-flux-1)\n\n### Background - Flux.1[#](#background-flux-1)\n\nFlux.1 is the text-to-image diffusion model benchmark, new to AMD’s MLPerf Training submissions in the v6.1 round. It measures how quickly the 12-billion-parameter Flux.1 Schnell model can be trained to reach the target validation loss (≤ 0.586) on the CC12M dataset, using 32 GPUs (4 nodes × 8 MI355X GPUs). The model has 19 double-stream (MMDiT) layers and 38 single-stream layers, with a hidden size of 3,072 and 24 attention heads.\n\nThis is one benchmark with two training recipes, which differ in the precision of the forward GEMMs of the transformer’s linear layers. Both recipes run all backward GEMMs in MXFP4:\n\n- **a6w4 + fp4 recipe:** the forward GEMMs use the a6w4 format. It runs from the published image`rocm/amd-mlperf:mlperf_flux1_6.1` .\n- **fp6 + fp4 recipe:** runs the forward GEMMs in MXFP6 on Primus’s Megatron-LM backend, while attention, normalization and communication run in BF16. It has no published image, so Step 1 builds it.\n\nPick one recipe and follow its steps; they are independent, and each has its own code folder, data, and results. The a6w4 + fp4 recipe needs about 2.5 TB on shared storage, plus about 2.5 TB of local disk per node. The fp6 + fp4 recipe needs about 2.4 TB on shared storage.\n\n### Recipe A: a6w4 + fp4[#](#recipe-a-a6w4-fp4)\n\n#### Step 1: Pull the Docker Image and Copy the Code - Flux.1 (a6w4 + fp4)[#](#step-1-pull-the-docker-image-and-copy-the-code-flux-1-a6w4-fp4)\n\n```\nmkdir -p /shared/mlperf_flux1 && cd /shared/mlperf_flux1\n\ndocker pull rocm/amd-mlperf:mlperf_flux1_6.1\n\ncontainer_id=$(docker create rocm/amd-mlperf:mlperf_flux1_6.1) && \\\ndocker cp $container_id:/workspace/code ./flux1_code && \\\ndocker rm $container_id\n```\n\nThe job runs its scripts from `flux1_code` on every node, so `/shared/mlperf_flux1` must be shared storage that every node can reach; replace it with your own shared path in every Flux.1 command. Every node also needs the image. Pull it on each node, for example with `srun --nodes=4 --ntasks-per-node=1 docker pull rocm/amd-mlperf:mlperf_flux1_6.1` (add your usual `--partition` option). To build the image instead, see the **Build the Images Instead of Pulling** section at the end of this guide.\n\n#### Step 2: Prepare Dataset - Flux.1 (a6w4 + fp4)[#](#step-2-prepare-dataset-flux-1-a6w4-fp4)\n\nDownload the preprocessed data to shared storage, running the script from the `flux1_code` folder that you copied in Step 1:\n\n```\ncd /shared/mlperf_flux1/flux1_code\n\nPREPROCESSED_PATH=/shared/mlperf_flux1/data ./scripts/download_ref_data.sh\n```\n\nThe data directory must contain the three downloaded directories, which training reads as-is:\n\n```\n/shared/mlperf_flux1/data/\n├── cc12m_preprocessed\n├── coco_preprocessed\n└── empty_encodings\n```\n\nThe script `job.sh` copies the data to `/data/mlperf_flux1` on each node (override with `FLUX_LOCAL_DATADIR`), so `DATADIR` must be set to a different path from `/data/mlperf_flux1`.\n\n#### Step 3: Set Up Environment - Flux.1 (a6w4 + fp4)[#](#step-3-set-up-environment-flux-1-a6w4-fp4)\n\nEnter the `flux1_code` folder that you copied in Step 1, then set the paths and the image:\n\n```\ncd /shared/mlperf_flux1/flux1_code\n\nmkdir -p /shared/mlperf_flux1/results\n\nexport DATADIR=/shared/mlperf_flux1/data\nexport LOGDIR=/shared/mlperf_flux1/results\nexport CONT=rocm/amd-mlperf:mlperf_flux1_6.1\n\nsudo chmod -R 777 $LOGDIR\n```\n\n`LOGDIR` must also be on shared storage. `sudo chmod -R 777 $LOGDIR` lets the container write its logs there.\n\nSource the 4-node (32-GPU) MI355X configuration:\n\n```\nsource config_MI355X_4x8x32.sh\n```\n\nThe `source` command loads the settings, including `NNODES`, into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.\n\n#### Step 4: Run Training - Flux.1 (a6w4 + fp4)[#](#step-4-run-training-flux-1-a6w4-fp4)\n\nEdit the `#SBATCH` header in `job.sh` to set your partition (`sinfo` lists the partitions you can use; replace `<your_partition>` with one of them), and uncomment `--nodelist` if you need specific nodes:\n\n```\ncd /shared/mlperf_flux1/flux1_code\nsed -i 's/--partition=YOUR_PARTITION/--partition=<your_partition>/' job.sh\n```\n\nSubmit from the `flux1_code` folder, in the same shell where you sourced the configuration, so that the job inherits `NNODES` and the other settings. Run one trial first to confirm that everything works:\n\n```\nexport NEXP=1\nsbatch --nodes=$NNODES job.sh\n```\n\n`sbatch` prints `Submitted batch job <jobid>` and returns at once; the job keeps running if you log out. Check its state with `squeue -u $USER`.\n\nFor a compliant 10-trial submission:\n\n```\nexport NEXP=10\nsbatch --nodes=$NNODES job.sh\n```\n\nThe job runs in three stages:\n\n1. **Stage the dataset.** Each node copies`cc12m_preprocessed` ,`coco_preprocessed` , and`empty_encodings` from`DATADIR` to its local disk at`/data/mlperf_flux1` (override with`FLUX_LOCAL_DATADIR` ), then writes a marker that records the source. Files already present with the same size are skipped, so only the first job copies the full ~2.5 TB. Training refuses to start on a node without a complete copy, so`FLUX_STAGE_LOCAL=0` , which skips this stage, only works when every node is already staged.\n2. **Train.** Each node starts one container, runs`runtime_tunables.sh` to tune the host, checks that an RDMA device is usable, and runs the`NEXP` trials back to back in that container.\n3. **Record.** After each trial, the first node writes that trial’s MLPerf records to`$LOGDIR/<DATESTAMP>_<N>.log` , numbered from 0, and runs the compliance checker.\n\nSlurm writes the job’s console output to `flux1_primus_<jobid>.out` and `flux1_primus_<jobid>.err` in the submission directory. The script `job.sh` requests a 60-hour time limit, which covers staging plus 10 trials of about 54 minutes each.\n\n#### Step 5: Check Quality and Validate Results - Flux.1 (a6w4 + fp4)[#](#step-5-check-quality-and-validate-results-flux-1-a6w4-fp4)\n\nQuality is the diffusion validation loss on 29,696 COCO-2014 samples (lower is better), evaluated every 262,144 training samples. A trial succeeds at the first evaluation at or below **0.586**.\n\nPrint the trial’s start, evaluation, and stop records:\n\n```\ngrep -E '\"key\": \"(run_start|eval_accuracy|run_stop)\"' \"$LOGDIR\"/*_0.log\n```\n\nThe trial converged if the last `eval_accuracy` value is 0.586 or lower and `run_stop` shows `\"status\": \"success\"`. Its time to train is the `run_stop` `time_ms` minus the `run_start` `time_ms`, in milliseconds. In AMD’s 4-node runs, converged trials met the target after 33 to 35 evaluations (8,650,752–9,175,040 samples), taking 47 to 50 minutes.\n\nTo score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight. Then run the RCP checker, after copying each trial log to the `result_<N>.txt` name it reads:\n\n```\ni=0; for f in \"$LOGDIR\"/*_[0-9]*.log; do cp \"$f\" \"$LOGDIR/result_$((i++)).txt\"; done\npython3 -m mlperf_logging.rcp_checker \"$LOGDIR\" --rcp_version 6.1.0\n```\n\nFinally, confirm that every trial’s section of `$LOGDIR/compliance_*.out` ends with `INFO - SUCCESS`.\n\n### Recipe B: fp6 + fp4[#](#recipe-b-fp6-fp4)\n\nThe network settings are written for the AMD Pensando™ Pollara 400 AI NICs (RoCE v2, one NIC per GPU) that AMD’s runs used; Step 3 shows where to change them for other RDMA NICs.\n\n#### Step 1: Get the Code and Build the Docker Image - Flux.1 (fp6 + fp4)[#](#step-1-get-the-code-and-build-the-docker-image-flux-1-fp6-fp4)\n\nThis recipe has no published image, so build it from the submission code in the MLCommons results repository. Copy the implementation folder to shared storage:\n\n```\nmkdir -p /shared/mlperf_flux1_fp6 && cd /shared/mlperf_flux1_fp6\n\ngit clone https://github.com/mlcommons/training_results_v6.1.git\ncp -r training_results_v6.1/AMD/benchmarks/flux1/implementations/4xMI355X_EPYC_9575F_primus_megatron ./flux1_fp6_code\n```\n\nThe job runs its scripts from `flux1_fp6_code` on every node, so `/shared/mlperf_flux1_fp6` must be on shared storage that every node can reach; replace it with your own shared path in every command below.\n\nEvery node needs the image. Build it on all four nodes at once (add your usual `--partition` option):\n\n```\nsrun --nodes=4 --ntasks-per-node=1 \\\n    bash -c 'cd /shared/mlperf_flux1_fp6/flux1_fp6_code && docker build -t mlperf_flux1_fp6:6.1 .'\n```\n\nThe `Dockerfile` starts from the public `rocm/primus:v26.7` image and installs AITER, Primus-Turbo and Primus at pinned commits. The build does not need a GPU. It also compiles the AITER kernels that training uses into the image, so nothing is compiled during a trial. The image records the commits it was built from in `/opt/mlperf/stack.txt`.\n\n#### Step 2: Prepare Dataset - Flux.1 (fp6 + fp4)[#](#step-2-prepare-dataset-flux-1-fp6-fp4)\n\nRun the following commands to download the dataset, convert it to the format training reads, and verify it:\n\n```\nmkdir -p /shared/mlperf_flux1_fp6/data\n\ndocker run --rm -v /shared/mlperf_flux1_fp6/data:/data mlperf_flux1_fp6:6.1 \\\n    bash /workspace/code/scripts/prepare_data.sh\n\ndocker run --rm -v /shared/mlperf_flux1_fp6/data:/data mlperf_flux1_fp6:6.1 \\\n    bash /workspace/code/scripts/verify_data.sh /data/mlperf_flux1\n```\n\nThe script `prepare_data.sh` downloads the MLCommons-preprocessed CC12M training set, the COCO-2014 evaluation subset and the empty encodings, and converts them to [Megatron-Energon](https://github.com/NVIDIA/Megatron-Energon) WebDataset shards in one streaming pass. It downloads four files at a time, converts each into one shard and deletes it, and keeps at most six downloaded files (about 1.2 GB) on disk at once. So the peak disk use is the final dataset size plus about 1.2 GB, not a raw download plus a converted copy. If the script stops partway, run it again: it skips the shards that already exist. After it completes, the data directory should contain the following:\n\n- Training: `mlperf_flux1/train/` (4,762 shards, 1,099,776 samples)\n- Validation: `mlperf_flux1/val/` (129 shards, 29,696 samples)\n- Empty encodings: `mlperf_flux1/empty_encodings/`\n- Energon metadata: `mlperf_flux1/.nv-meta/`\n\nThe script `verify_data.sh` checks these counts, that every validation sample has its evaluation timestep, and the checksum of every shard against the manifest shipped with the code. It ends with `[verify_data] PASS`. If a shard fails its checksum, for example one cut short when an earlier run of `prepare_data.sh` stopped, delete that shard and run `prepare_data.sh` again.\n\nEvery node reads the data from shared storage at a rate of about 1.5 GB/s per node. If your shared storage is slower, copy `mlperf_flux1` to the same local path on every node and set `DATADIR` there in Step 3.\n\n#### Step 3: Set Up Environment - Flux.1 (fp6 + fp4)[#](#step-3-set-up-environment-flux-1-fp6-fp4)\n\nThe script `job.sh` sets the image and the paths itself, so set them in that file rather than in your shell. Enter the `flux1_fp6_code` folder that you copied in Step 1, then set the paths and the image:\n\n```\ncd /shared/mlperf_flux1_fp6/flux1_fp6_code\n\nmkdir -p /shared/mlperf_flux1_fp6/results\n\nexport LOGDIR=/shared/mlperf_flux1_fp6/results\n\nsed -i \\\n    -e 's|^export CONT=.*|export CONT=mlperf_flux1_fp6:6.1|' \\\n    -e 's|^export DATADIR=.*|export DATADIR=/shared/mlperf_flux1_fp6/data|' \\\n    -e \"s|^export LOGDIR=.*|export LOGDIR=$LOGDIR|\" \\\n    -e 's|^export CACHEDIR=.*|export CACHEDIR=/tmp/mlperf_flux1_fp6_cache|' \\\n    -e 's|--partition=YOUR_PARTITION|--partition=<your_partition>|' \\\n    -e '/^export CONT=/a export CONT_PULL=0' \\\n    job.sh\n\nsudo chmod -R 777 $LOGDIR\n```\n\nThe environment variable `LOGDIR` must be on shared storage, and `CACHEDIR` holds the compile caches on each node’s local disk. `CONT_PULL=0` makes the launcher use the image you built instead of pulling it. `sinfo` lists the partitions you can use. `job.sh` uses the 4-node (32-GPU) configuration `config_MI355X_04x08x32_mxfp6heal_hb128.sh`, so there is no configuration to source.\n\nThe network settings are in `config_common.sh`. If your nodes do not have eight Pollara AI NICs named `ionic_0` through `ionic_7`, set `NCCL_IB_HCA` to your RDMA devices (`ls /sys/class/infiniband` lists them) and `NCCL_IB_GID_INDEX` to their RoCE v2 GID index. If your site provides an RCCL topology file for the nodes, set `RCCL_TOPO_FILE` in `job.sh` to its path on the host.\n\n#### Step 4: Run Training - Flux.1 (fp6 + fp4)[#](#step-4-run-training-flux-1-fp6-fp4)\n\nRun one trial first to confirm that everything works. It takes about 52 minutes, including about 4 minutes of startup before the timed run begins:\n\n```\ncd /shared/mlperf_flux1_fp6/flux1_fp6_code\nNEXP=1 sbatch job.sh\n```\n\nFor a compliant 10-trial submission (about 9 hours), submit the job with:\n\n```\nsbatch job.sh\n```\n\n`sbatch` prints `Submitted batch job <jobid>` and returns at once; check the job’s state with `squeue -u $USER`. Slurm writes the job’s console output to `flux1_<jobid>.out` and `flux1_<jobid>.err` in the submission folder, and each node writes its console output for trial `<N>` to `$LOGDIR/MI355X_04x08x32_mxfp6heal_hb128_run<N>_node<rank>.log`. A trial is finished when that output shows `ENDING TIMING RUN` followed by a line of the form `RESULT,FLUX1,<seed>,<seconds>,AMD,<start time>`, the total run time including startup.\n\nNote\n\nBefore the first trial, the launcher runs `runtime_tunables.sh`, which logs the host settings and warns where they differ from the ones AMD’s runs used; it does not change them. Before every trial, it drops the page cache, with passwordless `sudo` or through the privileged container, and does not start the trial if it cannot.\n\n#### Step 5: Check Quality and Validate Results - Flux.1 (fp6 + fp4)[#](#step-5-check-quality-and-validate-results-flux-1-fp6-fp4)\n\nQuality is the diffusion validation loss on 29,696 COCO-2014 samples (lower is better), evaluated every 262,144 training samples. A trial succeeds at the first evaluation at or below **0.586**.\n\nThis submission writes each trial’s MLPerf log to `$LOGDIR/result_<N>.txt`, numbered from 0. Print the trial’s start, evaluation, and stop records:\n\n```\ngrep -E '\"key\": \"(run_start|eval_accuracy|run_stop)\"' \"$LOGDIR\"/result_0.txt\n```\n\nThe trial converged if the last `eval_accuracy` value is 0.586 or lower and `run_stop` shows `\"status\": \"success\"`. Its time to train is the `run_stop` `time_ms` minus the `run_start` `time_ms`, in milliseconds. In AMD’s 10 submitted trials, the target was met after 32 to 36 evaluations (8,388,608–9,437,184 samples), taking 45.0 to 50.3 minutes.\n\nTo score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored **47.11 minutes**. The logs already have the names the RCP checker reads, so run it on `$LOGDIR` directly:\n\n```\npython3 -m mlperf_logging.rcp_checker \"$LOGDIR\" --rcp_version 6.1.0\n```\n\nFinally, confirm that every trial’s compliance file ends with `INFO - SUCCESS`:\n\n```\ntail -n 1 \"$LOGDIR\"/raw/*_node0/compliance_*.out\n```\n\n## Benchmark 5: GPT-OSS-20B[#](#benchmark-5-gpt-oss-20b)\n\n### Background - GPT-OSS-20B[#](#background-gpt-oss-20b)\n\nGPT-OSS-20B is a 20-billion-parameter Mixture-of-Experts language model pretraining benchmark new to AMD’s MLPerf Training portfolio in v6.1, trained on the C4 dataset to a target log perplexity of ≤ 3.34. **Disk space required:** ~85 GB dataset + ~32 GB model directory.\n\n### Step 1: Pull the Docker Image and Copy the Code - GPT-OSS-20B[#](#step-1-pull-the-docker-image-and-copy-the-code-gpt-oss-20b)\n\n```\nmkdir -p ~/mlperf && cd ~/mlperf\n\ndocker pull rocm/amd-mlperf:gpt_oss_20b_training_6.1\n\ncontainer_id=$(docker create rocm/amd-mlperf:gpt_oss_20b_training_6.1) && \\\ndocker cp $container_id:/workspace/code ./gpt_oss_20b_code && \\\ndocker rm $container_id\n```\n\nTo build the image instead, see the **Build the Images Instead of Pulling** section at the end of this guide.\n\n### Step 2: Prepare Dataset - GPT-OSS-20B[#](#step-2-prepare-dataset-gpt-oss-20b)\n\nRun the following commands to download the dataset and tokenizer:\n\n```\nmkdir -p /data/gpt_oss_20b\ncd /data/gpt_oss_20b\n\nbash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \\\n    -d data https://training.mlcommons-storage.org/metadata/llama-3-1-8b-preprocessed-c4-dataset.uri\n\nbash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \\\n    -d model https://training.mlcommons-storage.org/metadata/llama-3-1-8b-tokenizer.uri\n```\n\nAfter the download is complete, the data directory should contain the following:\n\n- Training: `c4-train.en_6_text_document.bin` and`.idx`\n- Validation: `c4-validation-91205-samples.en_text_document.bin` and`.idx`\n\n### Step 3: Set Up Environment - GPT-OSS-20B[#](#step-3-set-up-environment-gpt-oss-20b)\n\nEnter the `gpt_oss_20b_code` folder that you copied in Step 1, then set the paths and the image:\n\n```\ncd ~/mlperf/gpt_oss_20b_code\n\nmkdir -p /data/gpt_oss_20b/results\n\nexport DATADIR=/data/gpt_oss_20b/data\nexport MODELDIR=/data/gpt_oss_20b/model\nexport LOGDIR=/data/gpt_oss_20b/results\nexport CONT=rocm/amd-mlperf:gpt_oss_20b_training_6.1\n\nsudo chmod -R 777 $LOGDIR\n```\n\nSource the MI355X configuration:\n\n```\nsource config_MI355X_1x8x1_tp1pp1ep1_gbs32.sh\n```\n\nThe `source` command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.\n\n### Step 4: Run Training - GPT-OSS-20B[#](#step-4-run-training-gpt-oss-20b)\n\nRun one trial first to confirm that everything works. It takes about 61 minutes, including about 7 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows `ENDING TIMING RUN` followed by `RESULT,GPT_OSS_20B,,<seconds>,AMD,<start time>`, the total run time in seconds including startup.\n\n```\nexport NEXP=1\nbash run_with_docker.sh\n```\n\nFor a compliant 10-run submission (about 10 hours):\n\n```\nexport NEXP=10\nbash run_with_docker.sh\n```\n\nLogs default to MLPerf records and run summaries. Set `MLPERF_VERBOSE_LOGS=1` for debugging.\n\n### Step 5: Check Quality and Validate Results - GPT-OSS-20B[#](#step-5-check-quality-and-validate-results-gpt-oss-20b)\n\nQuality is validation log perplexity on held-out C4 text (lower is better), evaluated every 12,288 training sequences. A trial succeeds at the first evaluation at or below **3.34**.\n\nPrint the trial’s start, evaluation, and stop records:\n\n```\ngrep -E '\"key\": \"(run_start|eval_accuracy|run_stop)\"' \"$LOGDIR\"/*_1.log\n```\n\nThe trial converged if the last `eval_accuracy` value is 3.34 or lower and `run_stop` shows `\"status\": \"success\"`. Its time to train is the `run_stop` `time_ms` minus the `run_start` `time_ms`, in milliseconds. Across the 10 submitted trials, the target was met at the 18th or 19th evaluation (221,184–233,472 samples), with times to train of 54.0–57.0 minutes.\n\nThe last throughput record gives the average training throughput; expect approximately **68 samples/second** on MI355X:\n\n```\ngrep overall_throughput \"$LOGDIR\"/*_1.log | tail -1\n```\n\nTo score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored **56.18 minutes**. Then run the RCP checker, after copying each trial log to the `result_<N>.txt` name it reads:\n\n```\ni=0; for f in \"$LOGDIR\"/*_[0-9]*.log; do cp \"$f\" \"$LOGDIR/result_$((i++)).txt\"; done\npython3 -m mlperf_logging.rcp_checker \"$LOGDIR\" --rcp_version 6.1.0\n```\n\nFinally, confirm that every trial’s section of `$LOGDIR/compliance_*.out` ends with `INFO - SUCCESS`.\n\n## Troubleshooting[#](#troubleshooting)\n\nHere is a list of common issues and their resolutions:\n\n- **`permission denied` when running `docker`:** Your user is not in the` docker` group. Run`sudo usermod -aG docker $USER` , then log out and back in.\n- **`sudo: a password is required` in the console output:** The launch scripts cannot type a password. Configure passwordless`sudo` for your user, then start the run again.\n- **`cpupower: command not found`:** Install it with` sudo apt-get install linux-tools-common linux-tools-$(uname -r)` .\n- **`docker pull` fails with `unauthorized` or `pull access denied`:** Check that the image tag is spelled exactly as in the**Docker Images** table, and run`docker login` if your account needs it.\n- **A download stops partway:** Run the same download command again from the same folder. It resumes partial files and verifies checksums when it finishes.\n- **`No space left on device`:** Compare the sizes in**Disk Space, Time, and Where Things Live** with`df -h` on the Docker root folder and on`/data` . Free space, then start again.\n- **Llama2-70B LoRA download fails with 401 or 403:** The HuggingFace token is wrong, or the account has not accepted the license for`meta-llama/Llama-2-70b-hf` .\n- **The run ends within a minute or two:** Another job may be using the GPUs. Check`amd-smi monitor` and`docker ps` , and read the console output for the first error.\n- **The console is quiet for minutes:** This is normal. Startup takes several minutes, and training prints only when it evaluates the model. Check that the container is still running with`docker ps` .\n- **Starting over, or changing a script or configuration:** The launcher runs the copy in the code folder (`~/mlperf/<folder>` ,`/shared/mlperf_flux1/flux1_code` for the Flux.1 a6w4 + fp4 recipe, or`/shared/mlperf_flux1_fp6/flux1_fp6_code` for the Flux.1 fp6 + fp4 recipe), not the one inside the image, so edit that copy. To start from a clean copy, delete the folder with`sudo rm -rf` (the container writes files as root), then repeat Step 1. Repeating Step 1 into a folder that already exists nests a second copy inside it and keeps the old files.\n- **Stopping a run:** Press Ctrl+C, then find the container with`docker ps` and remove it with`docker rm -f <container name>` . For a Flux.1 job, run`scancel <jobid>` .\n- **Flux.1 stops at startup with an RDMA message:** The container could not open an RDMA device. Check`ls /dev/infiniband` on every node and that the nodes are connected to the RDMA network.\n\n## Expected Results[#](#expected-results)\n\nThe time-to-train values for the MLPerf Training v6.1 submissions by AMD are included below for reference. You should achieve similar results by following the procedures described above. Keep in mind that slight variations in hardware setups and conditions may lead to minor discrepancies from AMD’s results.\n\n| Submission ID | Model | MI355X TTT (min) | \n|---|---|---|\n| 6.1-0062 | Llama3.1-8B Pretraining | 72.94 | \n| 6.1-0062 | Llama2-70B LoRA Fine-Tuning | 6.98 | \n| 6.1-0061 | DLRMv4 | 18.95 | \n| 6.1-0062 | GPT-OSS-20B | 56.18 | \n| 6.1-0064 | Flux.1, a6w4 + fp4 recipe (4 nodes, 32 GPUs) | 47.18 | \n| 6.1-0001 | Flux.1, fp6 + fp4 recipe (4 nodes, 32 GPUs) | 47.11 | \n\n## Summary[#](#summary)\n\nThis guide covers the full reproduction workflow for AMD’s MLPerf Training v6.1 submission across five benchmarks — Llama3.1-8B pretraining, Llama2-70B LoRA fine-tuning, DLRMv4, Flux.1 (two recipes, a6w4 + fp4 and fp6 + fp4), and GPT-OSS-20B — on AMD Instinct MI355X GPUs. All benchmarks follow the same three-phase pattern:\n\n1. **Pull** the Docker image and copy the code out of it (the Flux.1 fp6 + fp4 recipe builds its image from the MLCommons code instead).\n2. **Prepare** the dataset and source the GPU-specific configuration file.\n3. **Run and check** each trial’s quality and time to train from its MLPerf log, then score the 10-trial set and run the MLCommons RCP and compliance checkers.\n\nThis round marks a significant expansion of AMD’s MLPerf Training portfolio — from three benchmarks in v6.0 to five in v6.1 — covering language model pretraining and fine-tuning, recommendation systems, and generative image modeling. Dive into the [MLPerf Training v6.1 blog](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-training-v6.1/README.html) to understand the optimizations behind these submissions, then put them into practice on AMD Instinct™ GPUs with ROCm.\n\n## Appendix: Build the Docker Images Instead of Pulling[#](#appendix-build-the-docker-images-instead-of-pulling)\n\nAMD’s submission code is also published in the MLCommons results repository. Each implementation folder there contains its Dockerfile and every file that the Dockerfile copies, so you can build an image from it instead of pulling one:\n\n```\ngit clone https://github.com/mlcommons/training_results_v6.1.git\ncd training_results_v6.1/AMD\n```\n\nRun the build from that benchmark’s implementation folder, and tag it with the image name from the **Docker Images** table so that `CONT` and the other commands in this guide work unchanged. If you build an image, skip the `docker pull` in that benchmark’s Step 1 and copy the code from your image with the same `docker create` and `docker cp` commands, or run the benchmark from the implementation folder itself.\n\n| Model | Implementation folder | Build command | \n|---|---|---|\n| Llama3.1-8B | `benchmarks/llama31_8b/implementations/MI355X_EPYC_9575F_primus` | `docker build -t rocm/amd-mlperf:llama31_8b_training_6.1 -f Dockerfile .` | \n| Llama2-70B LoRA | `benchmarks/llama2_70b_lora/implementations/MI355X_EPYC_9575F_primus` | `docker buildx build --network=host --load -t rocm/amd-mlperf:llama2_70b_training_6.1 .` | \n| DLRMv4 | `benchmarks/dlrmv4/implementations/MI355X_EPYC_9575F_hstu` | `docker build --network=host -t rocm/amd-mlperf:dlrmv4_training_6.1 -f Dockerfile .` | \n| Flux.1, a6w4 + fp4 recipe | `benchmarks/flux1/implementations/4xMI355X_EPYC_9575F_primus` | `DOCKER_BUILDKIT=1 docker build -t rocm/amd-mlperf:mlperf_flux1_6.1 -f Dockerfile .` | \n| Flux.1, fp6 + fp4 recipe | `benchmarks/flux1/implementations/4xMI355X_EPYC_9575F_primus_megatron` | `docker build -t mlperf_flux1_fp6:6.1 .` (Step 1 already builds it) | \n| GPT-OSS-20B | `benchmarks/gpt_oss_20b/implementations/MI355X_EPYC_9575F_primus` | `docker build -t rocm/amd-mlperf:gpt_oss_20b_training_6.1 .` | \n\nThe Dockerfiles pin every dependency to a fixed commit, and some of them are compiled from source:\n\n- **Llama2-70B LoRA** compiles Transformer Engine and Primus-Turbo, which takes a while. Run`sudo chmod -R 777 .` in the implementation folder before building so that all scripts are writable.\n- **DLRMv4** compiles FBGEMM-GPU, because there is no public gfx950 build. This takes 30–60 minutes;`--build-arg MAX_JOBS=<n>` sets the compile parallelism.\n- **Flux.1** runs on every node in the Slurm job, so each node needs the image. Either build it on every node, or push it to a registry that all nodes can reach and set`CONT` to that name.\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED “AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, AMD Instinct, EPYC, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/reproducing-amd-mlperf-training-v6-1-submission-result", "canonical_source": "https://rocm.blogs.amd.com/artificial-intelligence/mlperf-training6.1-repro/README.html", "published_at": "2026-11-16 00:00:00+00:00", "updated_at": "2026-11-18 19:21:48.986583+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-infrastructure", "ai-chips", "large-language-models"], "entities": ["AMD", "MLPerf Training v6.1", "AMD Instinct MI355X", "AMD Instinct MI350X", "Primus", "Llama3.1-8B", "Llama2-70B", "Flux.1"], "also_reported_by": [], "takeaway": "AMD's v6.1 guide lists the convergence targets and hardware each submission must hit to reproduce, including log perplexity ≤ 3.3 for Llama3.1-8B on C4 with 8 MI355X GPUs and validation loss ≤ 0.586 for Flux.1 on CC12M across 32 GPUs.", "recommendation": {"advice": "To reproduce AMD's MLPerf v6.1 runs, provision 8 MI355X GPUs per node on Ubuntu 22.04 with ROCm tools, idle GPUs and passwordless sudo, and use 4 Slurm nodes for the 32-GPU Flux.1 recipes.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/reproducing-amd-mlperf-training-v6-1-submission-result", "markdown": "https://wpnews.pro/news/reproducing-amd-mlperf-training-v6-1-submission-result.md", "text": "https://wpnews.pro/news/reproducing-amd-mlperf-training-v6-1-submission-result.txt", "jsonld": "https://wpnews.pro/news/reproducing-amd-mlperf-training-v6-1-submission-result.jsonld"}}