{"slug": "gpu-p2p-on-vs-off-tensor-parallel-llm-serving-on-four-r9700s", "title": "GPU P2P On vs Off: Tensor Parallel LLM Serving on Four R9700s", "summary": "Disabling peer-to-peer (P2P) memory access on four AMD Radeon AI PRO R9700 GPUs serving Qwen3.8-27B with tensor parallelism cut prefill throughput by 45% to 55%, decode by about 30%, and total output throughput with 16 clients by about 53%, according to a benchmark run by the blog's author using radiance 1.4.0 in MXFP4 at TP4 with each card capped at 250 W. With P2P off, median time to first token under 16-client load rose from 2.18 s to 7.93 s, and output tokens per joule across the four cards fell from 0.61 to 0.39 even though average power dropped from 985 W to 739 W. A 1 GB copy between two R9700s ran at 14.0 GB/s with P2P on, and GSM8K accuracy on 250 questions stayed within one question (98.8% versus 99.2%) while taking 80% longer.", "body_md": "# GPU P2P On vs Off: Tensor Parallel LLM Serving on Four R9700s\n\nMy four AMD Radeon AI PRO R9700s live in [the box with six GPUs in one x16\nslot](https://doug.sh/posts/six-gpus-one-x16-slot/), where they serve Qwen3.8-27B split across all four with tensor\nparallelism. Peer-to-peer (P2P) lets the cards read and write each other’s memory without a trip through\nthe CPU. On the R9700s it needs a 32 GB BAR on each card, the window the CPU uses to reach the card’s\nmemory. Getting those behind the switch took [a kernel\nmodule](https://github.com/dougcalobrisi/gpu-rebar-layout), and [the BARs\npost](https://doug.sh/posts/gpu-bars-behind-a-pcie-switch/) covers it.\n\nA 1 GB copy between two R9700s ran at 14.0 GB/s with P2P on. To see what P2P does for serving, I ran the same benchmark three times: P2P on, P2P off, and P2P on again to check the first run.\n\nWith P2P off, prefill dropped 45 to 55%, decode about 30%, and throughput with 16 clients about 53%. Median time to first token under load more than tripled.\n\n## Tensor parallelism and all-reduce [🔗](#tensor-parallelism-and-all-reduce)\n\nTensor parallelism splits each layer’s weights across the cards, so every card computes part of every layer for every token. Before the next layer can start, every card needs the sum of those partial results. That’s an all-reduce: each card sends its piece and gets the total back. In Megatron-LM’s layout there are two per layer, one after attention and one after the MLP.\n\nWith P2P on, the R9700s swap those partial sums directly through the switch. With P2P off, the sums go up to pinned host memory and back.\n\n## Running radiance with P2P on and off [🔗](#running-radiance-with-p2p-on-and-off)\n\nThe engine is [radiance](https://codeberg.org/StillDeadcode/radiance) 1.4.0 serving Qwen3.8-27B in MXFP4\nat tensor parallel 4 (TP4), with each card capped at 250 W. These flags stayed the same across the three\nruns, apart from the last line.\n\n```\n--max-num-seqs 16\n--max-model-len 262144\n--kv-cache-dtype fp8\n--tp-wire wht6\n--tp-wire-min-kb 128\n--p2p off    # the middle run only\n```\n\nWith `--p2p off`, radiance’s startup log says all-reduces go through host memory. Its graph debug output\nalso shows host all-reduce kernels replacing the peer ones, so turning P2P off changes both the code\nand the route.\n\n## Results: P2P off vs on [🔗](#results-p2p-off-vs-on)\n\n| Measurement | P2P on (a) | P2P off | P2P on (b) | \n|---|---|---|---|\n| Prefill tok/s, 2k prompt | 4,562 | 2,069 | 4,579 | \n| Prefill tok/s, 32k prompt | 4,674 | 2,142 | 4,690 | \n| Prefill tok/s, 128k prompt | 3,973 | 2,001 | 3,969 | \n| Prefill tok/s, 234k prompt (233,525 tokens) | 3,340 | 1,834 | 3,337 | \n| Decode tok/s, 2k context | 293.3 | 207.5 | 304.2 | \n| Decode tok/s, 234k context | 253.0 | 167.9 | 228.9 | \n| Total output tok/s, 1 client | 151.1 | 100.5 | 154.3 | \n| Total output tok/s, 16 clients | 601.1 | 285.7 | 608.1 | \n| Median time to first token, 16 clients | 2.18 s | 7.93 s | 2.38 s | \n| Longest time to first token, 16 clients | 3.49 s | 10.06 s | 3.48 s | \n| GSM8K accuracy, 250 questions | 98.8% | 99.2% | 98.8% | \n| Output tokens per joule, 4 cards, 16 clients | 0.61 | 0.39 | 0.62 | \n| Average power, 4 cards, 16 clients | 985 W | 739 W | 988 W | \n\nThe chart shows what was left, so the 2k prompt at 45% is a 55% drop. Short prompts lost the most prefill: 55% at 2k against 45% at 234k. The 233,525-token prompt waited 127.31 seconds for its first token with P2P off and 69.92 with it on. GSM8K, 250 grade-school math problems, came out one question apart and took 80% longer.\n\nThe two P2P-on runs were within 4% of each other on throughput, except long-context decode at about 10%, and median time to first token was 9% apart.\n\nWith P2P off the four cards drew 739 W instead of about 985 W, but throughput fell further, so GPU energy per output token rose about 59%. That counts only the power the four cards report.\n\n## Copy test vs all-reduce [🔗](#copy-test-vs-all-reduce)\n\nThe copy test moved one big buffer between one pair of cards. An all-reduce on four cards has every card sending and receiving at once.\n\nWith all four R9700s copying around a ring at once, the switch moved 54.7 GB/s. The host path carries about 28 GB/s each way over the x16 uplink to the CPU. All four cards share it, so they get about half that between them.\n\n## Checking P2P on your build [🔗](#checking-p2p-on-your-build)\n\n1. Benchmark your serving workload with P2P on and off, in addition to a copy test.\n2. Run P2P on twice, with short and long prompts, and one client and many.\n3. Check whether card-to-card traffic stays on a switch or goes through the CPU.\n4. Check that your driver supports P2P on your cards before you buy hardware for it.\n\n*I had help with this one. Claude ran the benchmarks on the box over SSH and helped me draft this post. I\ndid the physical work, read all the words, checked the numbers, and rewrote anything that sounded like a\nchatbot, so the mistakes are mine.*\n\n## Sources [🔗](#sources)\n\n- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053) , two all-reduces per transformer layer, after attention and after the MLP\n- [radiance](https://codeberg.org/StillDeadcode/radiance) , the inference engine on the R9700s", "url": "https://wpnews.pro/news/gpu-p2p-on-vs-off-tensor-parallel-llm-serving-on-four-r9700s", "canonical_source": "https://doug.sh/posts/gpu-p2p-tensor-parallel/", "published_at": "2026-11-12 02:30:00+00:00", "updated_at": "2026-11-12 15:20:50.786691+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "large-language-models", "mlops"], "entities": ["AMD", "Radeon AI PRO R9700", "Qwen3.8-27B", "radiance", "Megatron-LM", "GSM8K"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gpu-p2p-on-vs-off-tensor-parallel-llm-serving-on-four-r9700s", "markdown": "https://wpnews.pro/news/gpu-p2p-on-vs-off-tensor-parallel-llm-serving-on-four-r9700s.md", "text": "https://wpnews.pro/news/gpu-p2p-on-vs-off-tensor-parallel-llm-serving-on-four-r9700s.txt", "jsonld": "https://wpnews.pro/news/gpu-p2p-on-vs-off-tensor-parallel-llm-serving-on-four-r9700s.jsonld"}}