EP V1 vs V2 Inter-node Performance (DeepSeek-R1 on MI308X, MI300X and MI355X)
This compares the inter-node dispatch/combine performance of MORI EP V1 and V2 for the DeepSeek-R1 shape on three GPUs: MI308X, MI300X and MI355X. Section 1 is the test setup, section 2 the overall results, and section 3 the per-bucket data for each GPU.
1. Test setup
1.1 Common settings
Shape: 256 experts, top-8 per token, hidden 7168. EP16: two nodes × 8 GPUs.
Workload: uniform routing, buffer capacity equal to each call’s token count, no scales, no quantization. Combine is BF16; dispatch is measured in both FP8 and BF16. FP8 is OCP e4m3 on MI355X and e4m3 fnuz on MI308X and MI300X. T is the number of tokens per rank per call, from 4 to 4096.
Kernels: for T ≤ 512 the LL kernels are compared (V1 InterNodeV1LL vs V2 InterNodeV2LL), and for larger T the normal kernels (V1 InterNodeV1 vs V2 InterNodeV2), matching the switch point of V2’s
internode_kernel="auto". Every row compares kernels of the same family.V1: SHMEM backend with
num_qp_per_pe2. That is the default of the V1 tuning script (examples/ops/dispatch_combine/test_dispatch_combine_internode.py), and V1’s tuning tables were tuned with it; the V1 API default is 1. Geometry is chosen from V1’s tuning tables byMORI_EP_LAUNCH_CONFIG_MODE=AUTO; the rules actually used on each GPU are listed in section 3.V2: HIP backend with the default
num_qp_per_peof 8. The geometry and each bucket’s active QP count come from the tuning tables inpython/mori/ops/dispatch_combine_v2/tuning_configs/.Timing: both sides use the same data and correctness checks. Each measurement is 3 passes × (20 warmup + 30 timed rounds), dropping the first round of each pass, and latency is the mean. V1 and V2 are measured alternately three times, and each cell is the median of the three; runs that stalled throughout were discarded or re-measured. The D+C change compares the medians of each run’s D+C latency, not the sum of the D and C medians shown, so it can differ from a change computed from those cells; the largest difference is MI308X FP8 T4096 (−22.4% here, −25.1% from the D and C cells). Every run used here passed the correctness checks.
Bandwidth: as in V1, the mean over ranks and rounds of “bytes / time”, not the NIC line rate. The two variants differ only in how bytes are counted:
XGMI bandwidth counts the tokens this rank received.
RDMA bandwidth counts token × destination node, the local node included; for LL every token counts all nodes. With two nodes, about half of it actually crosses nodes.
Both sides move the same bytes in a cell, so a bandwidth change roughly mirrors the latency change. It is not the exact inverse, since bandwidth is a mean of per-round bytes / time and each cell is its own median; in two MI308X cells (FP8 T4096 combine, BF16 T2048 dispatch) both changes are negative, so the latency change favors V2 and the bandwidth change V1. In every row the XGMI and RDMA changes differ by less than 1.5 percentage points.
1.2 Environment per GPU
MI308X |
MI300X |
MI355X |
|
|---|---|---|---|
GPU |
AMD Instinct MI308X |
AMD Instinct MI300X |
AMD Instinct MI355X |
NIC (8 per node, 400 Gb/s) |
Broadcom BCM57608, RoCE |
NVIDIA ConnectX-7 (MT2910), InfiniBand |
AMD Pensando Pollara 400 (ionic), RoCE |
ROCm |
7.2.1, with the HIP runtime built separately from CLR source |
7.14 (HIP 7.14.60850) |
7.14 (HIP 7.14.60850) |
NIC firmware |
235.2.40.0 |
28.40.1702 |
1.117.5-a-45 |
The MI308X numbers were measured with a HIP runtime built from CLR source, not
the one ROCm 7.2.1 ships: with the stock 7.0–7.2.x runtime V2 cannot set up
across two nodes (hipMemSetAccess fails on its symmetric-memory windows; see
Issue 2 of the known-issues skill, .claude/skills/known-issues/SKILL.md).
ROCm 7.14 and later need no rebuild.
2. Overall results
How much lower V2’s D+C latency is than V1’s:
GPU |
dispatch |
small batch (T4–T32) |
medium batch (T64–T512) |
large batch (T1024–T4096) |
|---|---|---|---|---|
MI308X |
FP8 |
20%–31% |
11%–18% |
22%–28% |
MI308X |
BF16 |
18%–33% |
10%–16% |
9%–13% |
MI300X |
FP8 |
36%–42% |
12%–22% |
25%–38%; V1 runs borrowed rules, see 3.2 |
MI300X |
BF16 |
35%–43% |
10%–22% |
11%–22% |
MI355X |
FP8 |
23%–30% |
12%–19% |
7%–17% |
MI355X |
BF16 |
21%–30% |
13%–16% |
9%–15% |
Large-batch (T1024–T4096) bandwidth change (XGMI; the RDMA figures differ by less than 1.5 percentage points):
GPU |
dispatch |
dispatch bandwidth |
combine bandwidth |
|---|---|---|---|
MI308X |
FP8 |
+67% to +98% |
−4% to +15% |
MI308X |
BF16 |
−1% to +27% |
−10% to +20% |
MI300X |
FP8 |
+89% to +127%; V1 runs borrowed rules, see 3.2 |
+4% to +35% |
MI300X |
BF16 |
+14% to +20% |
+8% to +35% |
MI355X |
FP8 |
+11% to +18% |
+4% to +20% |
MI355X |
BF16 |
+10% to +14% |
+7% to +21% |
V2 is faster in every bucket on all three GPUs. No cell has a higher V2 D+C latency than V1.
The gain is largest at small batch. Small batches are 18%–43% faster; medium and large batches usually 7%–22%. FP8 large batches on MI308X and MI300X gain more (22%–38%), mostly in dispatch: on MI308X the two tuning tables themselves differ that much, and on MI300X V1 runs rules tuned for MI308X.
Small batches gain mostly in combine. At large batch it varies by GPU: FP8 on MI308X and MI300X gains mostly in dispatch; BF16 on MI300X and both dtypes on MI355X gain in both phases, by 8%–27% and 5%–18% respectively; MI308X’s combine is within 7% of V1 except at T2048.
Treat the MI308X small-batch numbers with caution. At T ≤ 32 V1 runs much slower than its own tuning table records; going by the two tables, V1 is the faster side at T ≤ 32, with V2’s FP8 D+C 3%–8% higher. See 3.1.
Bandwidth is listed for large batches only. Small batches are latency-bound, and their bandwidth change only mirrors the latency change.
Part of the gap comes from rules V1 never tuned. V1 has no BF16 dispatch rules for H7168 on MI308X and only some on MI355X, and V1’s MI300X table has only LL rules for T ≤ 64; the MI308X rules the normal kernel borrows are too small for MI300X. See section 3.
This is a kernel-level comparison, not end-to-end inference. Only this one shape with uniform routing was measured; real workloads route unevenly and run expert computation between dispatch and combine.
3. Per-GPU data
In the kernel column, LL means V1 InterNodeV1LL vs V2 InterNodeV2LL, and normal means V1 InterNodeV1 vs V2 InterNodeV2. D is dispatch, C is combine, and T is the number of tokens per rank per call. Latency is in µs and bandwidth in GB/s; bandwidth cells read “V1 → V2 (change)”. Change = V2 / V1 − 1, so a negative latency change and a positive bandwidth change mean V2 is better.
3.1 MI308X
dispatch |
T |
kernel |
V1 D / C |
V2 D / C |
D latency |
C latency |
D+C latency |
XGMI BW D |
XGMI BW C |
RDMA BW D |
RDMA BW C |
|---|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
4 |
LL |
60.0 / 93.0 |
47.0 / 67.8 |
−21.7% |
−27.1% |
−24.9% |
3.20 → 4.06 (+26.9%) |
4.11 → 5.63 (+37.0%) |
0.96 → 1.22 (+27.1%) |
1.24 → 1.69 (+36.3%) |
FP8 |
8 |
LL |
61.3 / 101.2 |
50.9 / 60.6 |
−16.9% |
−40.1% |
−31.4% |
6.15 → 7.39 (+20.2%) |
7.44 → 12.41 (+66.8%) |
1.88 → 2.26 (+20.2%) |
2.27 → 3.80 (+67.4%) |
FP8 |
16 |
LL |
61.9 / 103.8 |
48.7 / 69.0 |
−21.4% |
−33.5% |
−28.9% |
12.13 → 15.47 (+27.5%) |
14.46 → 21.72 (+50.2%) |
3.72 → 4.74 (+27.4%) |
4.43 → 6.66 (+50.3%) |
FP8 |
32 |
LL |
67.3 / 114.8 |
57.6 / 87.5 |
−14.5% |
−23.8% |
−20.3% |
22.32 → 26.30 (+17.8%) |
26.10 → 34.38 (+31.7%) |
6.85 → 8.07 (+17.8%) |
8.00 → 10.54 (+31.7%) |
FP8 |
64 |
LL |
85.9 / 153.0 |
77.0 / 121.7 |
−10.3% |
−20.5% |
−16.8% |
35.18 → 40.02 (+13.8%) |
39.34 → 49.77 (+26.5%) |
10.74 → 12.21 (+13.7%) |
12.01 → 15.21 (+26.6%) |
FP8 |
128 |
LL |
121.4 / 203.8 |
97.5 / 176.8 |
−19.7% |
−13.2% |
−15.6% |
50.76 → 62.66 (+23.4%) |
59.27 → 68.20 (+15.1%) |
15.59 → 19.23 (+23.3%) |
18.18 → 20.92 (+15.1%) |
FP8 |
256 |
LL |
186.2 / 304.1 |
153.8 / 280.8 |
−17.4% |
−7.7% |
−11.4% |
65.86 → 79.56 (+20.8%) |
79.45 → 85.87 (+8.1%) |
20.17 → 24.36 (+20.8%) |
24.33 → 26.30 (+8.1%) |
FP8 |
512 |
LL |
361.7 / 514.4 |
246.2 / 469.7 |
−31.9% |
−8.7% |
−18.2% |
67.22 → 97.58 (+45.2%) |
93.75 → 101.92 (+8.7%) |
20.63 → 29.96 (+45.2%) |
28.77 → 31.28 (+8.7%) |
FP8 |
1024 |
normal |
826.6 / 913.8 |
414.2 / 853.5 |
−49.9% |
−6.6% |
−27.1% |
60.13 → 118.95 (+97.8%) |
108.08 → 112.84 (+4.4%) |
18.36 → 36.30 (+97.7%) |
32.97 → 34.43 (+4.4%) |
FP8 |
2048 |
normal |
1531.6 / 1549.8 |
911.5 / 1317.7 |
−40.5% |
−15.0% |
−27.7% |
65.01 → 108.71 (+67.2%) |
127.32 → 146.99 (+15.4%) |
19.86 → 33.22 (+67.3%) |
38.89 → 44.90 (+15.5%) |
FP8 |
4096 |
normal |
3264.7 / 3167.7 |
1719.3 / 3098.9 |
−47.3% |
−2.2% |
−22.4% |
62.56 → 112.37 (+79.6%) |
128.47 → 123.82 (−3.6%) |
19.13 → 34.35 (+79.6%) |
39.29 → 37.85 (−3.7%) |
BF16 |
4 |
LL |
59.8 / 93.2 |
47.4 / 66.0 |
−20.6% |
−29.2% |
−25.7% |
6.42 → 8.05 (+25.4%) |
4.10 → 5.78 (+41.0%) |
1.93 → 2.43 (+25.9%) |
1.23 → 1.74 (+41.5%) |
BF16 |
8 |
LL |
62.1 / 101.0 |
49.7 / 60.0 |
−19.9% |
−40.6% |
−32.7% |
12.13 → 15.18 (+25.1%) |
7.44 → 12.53 (+68.4%) |
3.71 → 4.64 (+25.1%) |
2.28 → 3.83 (+68.0%) |
BF16 |
16 |
LL |
66.4 / 102.6 |
52.9 / 68.1 |
−20.4% |
−33.6% |
−28.4% |
22.61 → 28.82 (+27.5%) |
14.62 → 22.20 (+51.8%) |
6.94 → 8.83 (+27.2%) |
4.48 → 6.81 (+52.0%) |
BF16 |
32 |
LL |
78.7 / 114.4 |
70.8 / 87.8 |
−10.0% |
−23.3% |
−17.9% |
38.20 → 43.51 (+13.9%) |
26.21 → 34.70 (+32.4%) |
11.71 → 13.34 (+13.9%) |
8.04 → 10.64 (+32.3%) |
BF16 |
64 |
LL |
109.9 / 152.3 |
99.3 / 121.0 |
−9.7% |
−20.5% |
−16.0% |
55.32 → 61.00 (+10.3%) |
39.71 → 49.94 (+25.8%) |
16.90 → 18.63 (+10.2%) |
12.12 → 15.25 (+25.8%) |
BF16 |
128 |
LL |
168.6 / 204.2 |
144.8 / 177.2 |
−14.1% |
−13.2% |
−13.7% |
72.61 → 83.25 (+14.7%) |
59.60 → 68.27 (+14.5%) |
22.30 → 25.55 (+14.6%) |
18.28 → 20.94 (+14.6%) |
BF16 |
256 |
LL |
269.2 / 311.9 |
239.9 / 283.2 |
−10.9% |
−9.2% |
−10.2% |
91.34 → 100.94 (+10.5%) |
78.07 → 85.22 (+9.2%) |
27.96 → 30.91 (+10.6%) |
23.92 → 26.10 (+9.1%) |
BF16 |
512 |
LL |
497.5 / 507.4 |
418.6 / 469.6 |
−15.9% |
−7.4% |
−11.6% |
97.28 → 114.51 (+17.7%) |
95.23 → 101.94 (+7.0%) |
29.85 → 35.15 (+17.8%) |
29.24 → 31.29 (+7.0%) |
BF16 |
1024 |
normal |
923.9 / 874.8 |
713.1 / 837.8 |
−22.8% |
−4.2% |
−12.7% |
106.49 → 135.41 (+27.2%) |
112.56 → 114.85 (+2.0%) |
32.49 → 41.31 (+27.1%) |
34.35 → 35.05 (+2.0%) |
BF16 |
2048 |
normal |
1622.4 / 1616.5 |
1607.5 / 1321.6 |
−0.9% |
−18.2% |
−9.4% |
121.21 → 120.21 (−0.8%) |
121.53 → 146.26 (+20.3%) |
37.03 → 36.72 (−0.8%) |
37.12 → 44.68 (+20.4%) |
BF16 |
4096 |
normal |
3619.4 / 2957.5 |
2765.5 / 3110.2 |
−23.6% |
+5.2% |
−11.2% |
110.66 → 138.71 (+25.3%) |
136.87 → 123.31 (−9.9%) |
33.83 → 42.41 (+25.4%) |
41.84 → 37.70 (−9.9%) |
Where the gain comes from. Up to T64 the LL kernel gains mostly in combine, about 40% at T8; from T128 to T512 dispatch gains as much or more. The normal kernel gains mostly in dispatch, which is 40%–50% faster with FP8; BF16 T2048 is the exception, with dispatch level and combine 18% faster. The normal kernel’s combine is clearly faster only at T2048 (15%–18%); at T1024 and T4096 it is within 7% of V1.
V1 has no tuned rules for BF16 dispatch. V1’s tuning tables have no BF16 dispatch rule for H7168: V1’s LL kernel borrows the H6144 (GLM-5) rules by relaxed matching (the four for T ≤ 32, with larger T reusing the T32 rule), and the normal kernel uses the AUTO default 96/64/8. So part of the BF16 gap comes from V1 not being tuned; with FP8 both sides use their own tuned rules.
Data and stability. This is the cleanest of five identical measurements taken from 2026-11-03 to 11-05. V1’s per-cell spread across the five has a median of about 2%; V2 is as stable at T ≤ 512 and in combine. V2’s large-batch dispatch is not: in the 11-04 run FP8 T1024 and T2048 were 23%–32% faster than in the other four; FP8 T4096 occasionally has rounds whose mean latency is 2–4× higher, and in one of the five runs two of the three rounds stalled; BF16 T1024 and T2048 were 8%–17% slower on 11-05 than on 11-03 and 11-04. The cause has not been investigated.
V1 is slower at small batch than its own tuning table. At T ≤ 32, V1’s dispatch is 13%–30% slower than the table records and its combine 37%–67% slower, in all five measurements; at T64–T512 it is within ±10% of the table. V2 is within 7% of its table at T ≤ 512. Going by the two tables, V2’s FP8 D+C at T ≤ 32 is 3%–8% higher than V1’s (V1 has no H7168 rule for BF16 dispatch), so the small-batch gap here may come entirely from the measurement conditions; the cause has not been investigated. These V1 rules (#689) were measured with V1’s own benchmark script, keeping only the lowest-latency runs.
3.2 MI300X
dispatch |
T |
kernel |
V1 D / C |
V2 D / C |
D latency |
C latency |
D+C latency |
XGMI BW D |
XGMI BW C |
RDMA BW D |
RDMA BW C |
|---|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
4 |
LL |
45.7 / 77.7 |
37.2 / 40.3 |
−18.7% |
−48.1% |
−37.2% |
4.20 → 5.20 (+23.8%) |
4.92 → 9.50 (+93.1%) |
1.26 → 1.57 (+24.6%) |
1.48 → 2.86 (+93.2%) |
FP8 |
8 |
LL |
44.9 / 77.8 |
37.8 / 40.7 |
−15.7% |
−47.6% |
−36.0% |
8.42 → 10.10 (+20.0%) |
9.68 → 18.55 (+91.6%) |
2.58 → 3.09 (+19.8%) |
2.96 → 5.67 (+91.6%) |
FP8 |
16 |
LL |
45.0 / 89.4 |
35.0 / 43.2 |
−22.1% |
−51.7% |
−41.6% |
16.71 → 21.45 (+28.4%) |
16.79 → 34.72 (+106.8%) |
5.13 → 6.57 (+28.1%) |
5.14 → 10.65 (+107.2%) |
FP8 |
32 |
LL |
47.7 / 95.3 |
39.7 / 52.5 |
−16.7% |
−45.0% |
−35.5% |
31.52 → 37.87 (+20.1%) |
31.43 → 57.23 (+82.1%) |
9.67 → 11.61 (+20.1%) |
9.64 → 17.55 (+82.1%) |
FP8 |
64 |
LL |
56.4 / 106.3 |
50.9 / 76.8 |
−9.7% |
−27.7% |
−21.5% |
53.43 → 59.37 (+11.1%) |
56.61 → 78.47 (+38.6%) |
16.32 → 18.14 (+11.2%) |
17.29 → 23.96 (+38.6%) |
FP8 |
128 |
LL |
83.0 / 146.1 |
69.8 / 117.7 |
−15.8% |
−19.4% |
−18.0% |
72.48 → 87.17 (+20.3%) |
82.07 → 102.17 (+24.5%) |
22.22 → 26.73 (+20.3%) |
25.15 → 31.30 (+24.5%) |
FP8 |
256 |
LL |
110.7 / 214.1 |
106.0 / 178.4 |
−4.3% |
−16.7% |
−12.3% |
109.39 → 113.52 (+3.8%) |
112.29 → 134.58 (+19.9%) |
33.48 → 34.74 (+3.8%) |
34.36 → 41.18 (+19.8%) |
FP8 |
512 |
LL |
181.0 / 362.1 |
151.2 / 313.1 |
−16.4% |
−13.5% |
−14.5% |
137.20 → 160.34 (+16.9%) |
133.49 → 153.42 (+14.9%) |
42.00 → 49.09 (+16.9%) |
40.88 → 46.98 (+14.9%) |
FP8 |
1024 |
normal |
576.4 / 606.5 |
251.9 / 492.1 |
−56.3% |
−18.9% |
−37.1% |
83.86 → 190.53 (+127.2%) |
159.15 → 194.91 (+22.5%) |
25.60 → 58.16 (+127.2%) |
48.59 → 59.50 (+22.5%) |
FP8 |
2048 |
normal |
947.3 / 1159.1 |
458.4 / 848.9 |
−51.6% |
−26.8% |
−37.9% |
102.62 → 208.81 (+103.5%) |
166.73 → 225.45 (+35.2%) |
31.39 → 63.87 (+103.5%) |
50.99 → 68.95 (+35.2%) |
FP8 |
4096 |
normal |
1749.4 / 1756.7 |
930.3 / 1683.0 |
−46.8% |
−4.2% |
−25.4% |
109.45 → 206.64 (+88.8%) |
218.15 → 227.76 (+4.4%) |
33.46 → 63.18 (+88.8%) |
66.70 → 69.65 (+4.4%) |
BF16 |
4 |
LL |
62.4 / 78.2 |
42.1 / 40.0 |
−32.6% |
−48.9% |
−41.6% |
6.14 → 9.19 (+49.7%) |
4.89 → 9.60 (+96.3%) |
1.85 → 2.77 (+49.7%) |
1.47 → 2.89 (+96.6%) |
BF16 |
8 |
LL |
60.9 / 78.3 |
41.8 / 39.7 |
−31.4% |
−49.3% |
−41.2% |
12.36 → 18.24 (+47.6%) |
9.62 → 18.99 (+97.4%) |
3.78 → 5.57 (+47.4%) |
2.94 → 5.81 (+97.6%) |
BF16 |
16 |
LL |
62.0 / 88.3 |
42.2 / 43.1 |
−31.9% |
−51.2% |
−43.2% |
24.20 → 35.56 (+46.9%) |
16.98 → 34.84 (+105.2%) |
7.42 → 10.90 (+46.9%) |
5.21 → 10.68 (+105.0%) |
BF16 |
32 |
LL |
65.3 / 92.9 |
50.0 / 52.4 |
−23.5% |
−43.6% |
−35.3% |
45.90 → 60.12 (+31.0%) |
32.23 → 57.35 (+77.9%) |
14.07 → 18.43 (+31.0%) |
9.89 → 17.59 (+77.9%) |
BF16 |
64 |
LL |
81.1 / 106.4 |
68.6 / 77.4 |
−15.4% |
−27.2% |
−22.1% |
74.46 → 87.93 (+18.1%) |
56.73 → 77.89 (+37.3%) |
22.75 → 26.85 (+18.0%) |
17.32 → 23.78 (+37.3%) |
BF16 |
128 |
LL |
115.6 / 148.4 |
97.0 / 118.4 |
−16.1% |
−20.2% |
−18.5% |
104.02 → 124.53 (+19.7%) |
81.00 → 101.77 (+25.6%) |
31.89 → 38.17 (+19.7%) |
24.82 → 31.19 (+25.7%) |
BF16 |
256 |
LL |
167.9 / 218.1 |
145.8 / 177.7 |
−13.2% |
−18.5% |
−16.1% |
143.42 → 164.79 (+14.9%) |
110.16 → 135.06 (+22.6%) |
43.89 → 50.43 (+14.9%) |
33.71 → 41.33 (+22.6%) |
BF16 |
512 |
LL |
268.0 / 354.4 |
246.3 / 314.6 |
−8.1% |
−11.2% |
−9.9% |
179.45 → 195.51 (+8.9%) |
135.50 → 152.75 (+12.7%) |
54.94 → 59.86 (+9.0%) |
41.49 → 46.78 (+12.8%) |
BF16 |
1024 |
normal |
587.2 / 608.8 |
483.6 / 505.7 |
−17.6% |
−16.9% |
−17.3% |
166.27 → 198.61 (+19.5%) |
159.42 → 189.94 (+19.1%) |
50.75 → 60.62 (+19.4%) |
48.67 → 57.99 (+19.1%) |
BF16 |
2048 |
normal |
1020.7 / 1173.4 |
844.1 / 858.8 |
−17.3% |
−26.8% |
−22.3% |
191.29 → 226.85 (+18.6%) |
165.30 → 222.93 (+34.9%) |
58.52 → 69.38 (+18.6%) |
50.55 → 68.18 (+34.9%) |
BF16 |
4096 |
normal |
1951.8 / 1829.6 |
1692.5 / 1674.1 |
−13.3% |
−8.5% |
−10.9% |
198.32 → 226.68 (+14.3%) |
211.64 → 229.08 (+8.2%) |
60.63 → 69.31 (+14.3%) |
64.72 → 70.05 (+8.2%) |
Where the gain comes from. Small batches (LL) gain mostly in combine, 11%–52% across the buckets; large batches (normal kernel) gain mostly in dispatch with FP8 (47%–56%), and in both phases with BF16 (8%–27%).
System setting. NUMA auto-balancing was disabled (
kernel.numa_balancing=0). With it on, the kernel’s scans of process memory make KFD evict all of a process’s GPU queues for about 17 ms, and in EP16 every rank waits when any one rank is evicted. Both V1 and V2 are affected: mean latency rises noticeably and a table can show a false “V2 is slower”, while bandwidth and medians are less affected.For most buckets V1 has no rules tuned for MI300X. The configurations AUTO actually used:
LL FP8 dispatch: T4–T64 use the rules in the MI300X table (16/8/6, 64/32/4, 64/32/4, 128/64/4, 256/128/4 in order), and T128–T512 reuse the T64 rule (256/128/4).
LL BF16 dispatch: the table has no BF16 rules, so the AUTO default 256/128/8 is used.
LL combine: T4–T64 use the rules in the MI300X table (304/76/4, 304/76/4, 128/32/8, 64/42/8, 128/85/4 in order), and T128–T512 reuse the T64 rule (128/85/4).
Normal kernel (T1024–T4096): MI300X has no InterNodeV1 table, so AUTO borrows the table of MI308X, also gfx942. FP8 dispatch uses 32/16/8, 64/32/4 and 32/16/8 in order, and combine 80/40/8, 64/32/16 and 64/32/8; BF16 dispatch has no rules and uses the default 96/64/8. MI308X has only 80 CUs, so these rules are too small for the 304-CU MI300X: V1’s FP8 T1024 dispatch is 576 µs, against about 355 µs with the same 2 QPs and the default geometry 96/64/8. Much of the FP8 normal-kernel dispatch gap comes from this.
3.3 MI355X
dispatch |
T |
kernel |
V1 D / C |
V2 D / C |
D latency |
C latency |
D+C latency |
XGMI BW D |
XGMI BW C |
RDMA BW D |
RDMA BW C |
|---|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
4 |
LL |
46.4 / 65.9 |
36.2 / 44.3 |
−21.9% |
−32.8% |
−28.1% |
4.12 → 5.33 (+29.4%) |
5.78 → 8.66 (+49.8%) |
1.24 → 1.60 (+29.0%) |
1.74 → 2.61 (+50.0%) |
FP8 |
8 |
LL |
46.5 / 66.0 |
37.5 / 45.4 |
−19.3% |
−31.3% |
−26.6% |
8.10 → 10.20 (+25.9%) |
11.39 → 16.81 (+47.6%) |
2.48 → 3.12 (+25.8%) |
3.48 → 5.15 (+48.0%) |
FP8 |
16 |
LL |
49.5 / 78.8 |
39.8 / 49.6 |
−19.5% |
−37.1% |
−30.2% |
15.19 → 19.13 (+25.9%) |
19.02 → 30.48 (+60.3%) |
4.66 → 5.86 (+25.8%) |
5.83 → 9.35 (+60.4%) |
FP8 |
32 |
LL |
56.1 / 81.8 |
45.0 / 61.8 |
−19.7% |
−24.5% |
−22.6% |
26.86 → 33.70 (+25.5%) |
36.69 → 48.80 (+33.0%) |
8.24 → 10.33 (+25.4%) |
11.25 → 14.97 (+33.1%) |
FP8 |
64 |
LL |
70.5 / 110.2 |
62.6 / 87.1 |
−11.3% |
−20.9% |
−17.2% |
42.78 → 48.53 (+13.4%) |
54.66 → 69.34 (+26.9%) |
13.07 → 14.83 (+13.5%) |
16.69 → 21.17 (+26.8%) |
FP8 |
128 |
LL |
87.3 / 134.8 |
72.2 / 107.4 |
−17.4% |
−20.3% |
−19.1% |
69.92 → 84.37 (+20.7%) |
89.39 → 112.38 (+25.7%) |
21.42 → 25.87 (+20.8%) |
27.41 → 34.44 (+25.6%) |
FP8 |
256 |
LL |
114.4 / 192.1 |
99.3 / 164.0 |
−13.2% |
−14.6% |
−14.1% |
107.50 → 122.58 (+14.0%) |
125.76 → 146.98 (+16.9%) |
32.90 → 37.51 (+14.0%) |
38.48 → 44.98 (+16.9%) |
FP8 |
512 |
LL |
155.6 / 306.2 |
143.5 / 265.3 |
−7.8% |
−13.4% |
−11.5% |
155.43 → 169.23 (+8.9%) |
156.94 → 181.37 (+15.6%) |
47.59 → 51.82 (+8.9%) |
48.06 → 55.54 (+15.6%) |
FP8 |
1024 |
normal |
274.6 / 557.3 |
232.0 / 460.2 |
−15.5% |
−17.4% |
−16.8% |
180.60 → 207.48 (+14.9%) |
173.51 → 208.74 (+20.3%) |
55.07 → 63.27 (+14.9%) |
52.91 → 63.65 (+20.3%) |
FP8 |
2048 |
normal |
501.0 / 918.5 |
411.8 / 829.2 |
−17.8% |
−9.7% |
−12.6% |
197.90 → 233.91 (+18.2%) |
210.40 → 231.07 (+9.8%) |
60.58 → 71.61 (+18.2%) |
64.40 → 70.72 (+9.8%) |
FP8 |
4096 |
normal |
872.8 / 1650.5 |
765.8 / 1573.5 |
−12.3% |
−4.7% |
−7.3% |
227.16 → 251.85 (+10.9%) |
234.25 → 243.84 (+4.1%) |
69.39 → 76.92 (+10.9%) |
71.55 → 74.48 (+4.1%) |
BF16 |
4 |
LL |
53.7 / 66.6 |
40.9 / 43.0 |
−23.8% |
−35.4% |
−30.2% |
7.13 → 9.34 (+31.0%) |
5.73 → 8.88 (+55.0%) |
2.14 → 2.81 (+31.3%) |
1.73 → 2.68 (+54.9%) |
BF16 |
8 |
LL |
55.3 / 64.3 |
42.9 / 44.3 |
−22.4% |
−31.2% |
−27.1% |
13.62 → 17.64 (+29.5%) |
11.68 → 17.05 (+46.0%) |
4.16 → 5.39 (+29.6%) |
3.57 → 5.22 (+46.2%) |
BF16 |
16 |
LL |
58.9 / 77.7 |
48.7 / 49.8 |
−17.4% |
−35.9% |
−27.9% |
25.62 → 31.33 (+22.3%) |
19.34 → 30.55 (+58.0%) |
7.85 → 9.59 (+22.2%) |
5.93 → 9.38 (+58.2%) |
BF16 |
32 |
LL |
71.1 / 80.8 |
58.9 / 60.7 |
−17.1% |
−24.9% |
−21.4% |
42.47 → 51.32 (+20.8%) |
37.28 → 49.75 (+33.4%) |
13.02 → 15.73 (+20.8%) |
11.43 → 15.26 (+33.5%) |
BF16 |
64 |
LL |
95.9 / 109.4 |
86.6 / 86.7 |
−9.7% |
−20.8% |
−15.8% |
63.18 → 70.24 (+11.2%) |
55.25 → 70.37 (+27.4%) |
19.30 → 21.47 (+11.2%) |
16.87 → 21.47 (+27.3%) |
BF16 |
128 |
LL |
108.8 / 134.9 |
101.8 / 107.8 |
−6.4% |
−20.1% |
−14.0% |
112.97 → 119.35 (+5.6%) |
89.90 → 112.48 (+25.1%) |
34.61 → 36.58 (+5.7%) |
27.56 → 34.46 (+25.0%) |
BF16 |
256 |
LL |
183.0 / 193.1 |
152.8 / 164.7 |
−16.5% |
−14.7% |
−15.6% |
132.23 → 157.83 (+19.4%) |
125.25 → 146.44 (+16.9%) |
40.47 → 48.30 (+19.3%) |
38.33 → 44.81 (+16.9%) |
BF16 |
512 |
LL |
264.1 / 301.2 |
230.5 / 263.1 |
−12.7% |
−12.7% |
−12.7% |
182.19 → 209.60 (+15.0%) |
159.71 → 183.24 (+14.7%) |
55.79 → 64.18 (+15.0%) |
48.90 → 56.11 (+14.7%) |
BF16 |
1024 |
normal |
470.4 / 552.1 |
410.3 / 455.2 |
−12.8% |
−17.6% |
−15.4% |
206.32 → 235.02 (+13.9%) |
175.12 → 211.64 (+20.9%) |
62.91 → 71.67 (+13.9%) |
53.40 → 64.54 (+20.9%) |
BF16 |
2048 |
normal |
851.5 / 920.5 |
754.9 / 819.2 |
−11.3% |
−11.0% |
−11.2% |
227.92 → 254.03 (+11.5%) |
210.46 → 234.06 (+11.2%) |
69.76 → 77.76 (+11.5%) |
64.43 → 71.64 (+11.2%) |
BF16 |
4096 |
normal |
1609.2 / 1674.0 |
1447.9 / 1548.4 |
−10.0% |
−7.5% |
−8.7% |
241.96 → 265.72 (+9.8%) |
232.36 → 248.27 (+6.8%) |
73.91 → 81.15 (+9.8%) |
70.96 → 75.83 (+6.9%) |
Where the gain comes from. Small batches (LL) gain mostly in combine, 13%–37%, with dispatch 6%–24% faster; large batches (normal kernel) gain 5%–18% in both phases. The gap narrows as T grows; at T4096 D+C is 7%–9% faster.
For most buckets V1 uses the table’s rule for that bucket. Every bucket does for FP8 dispatch and BF16 combine. Of the buckets measured here, BF16 dispatch has rules only for LL T64 and T128 and normal-kernel T4096, and AUTO uses the next larger rule for the rest: LL T4–T32 use T64’s 128/64/8, LL T256 and T512 use the LL table’s T2048 rule (256/170/16), and normal-kernel T1024 and T2048 use T4096’s 256/128/16.
Both sides’ measurements match their own tuning tables. With the same rule, V1’s measurement in every bucket is within −6% to +13% of its table record, and V2’s within −10% to +1%.