MORI-IO Benchmark

Table of Contents

• Which benchmark to use

• C++ Benchmark (default, nixlbench-matching)

  • Build

  • Run (two nodes)

  • Sweep modes

  • Differences from the Python flags

• Python Benchmark

  • Fabric (cross-node scale-up UALink super-node)

• Benchmark Arguments

• Results: Thor2 RDMA Read

• Results: Thor2 RDMA Write

  • Message Size Sweep

  • Batch Size Sweep

• Results: CX7 RDMA (Batch Size = 1)

  • Write

  • Read

• Benchmark Tuning Guide

  • Recommended starting config

  • Parameters that help

  • The merged-WR size limit

  • Parameters that usually do not help (or can hurt)

  • Host (CPU) vs GPU memory

Which benchmark to use

MORI-IO ships two benchmarks that measure the same transfer paths:

• C++ (tests/cpp/io/bench_engine) — the default. No Python interpreter in the measurement loop, and its timing matches nixlbench so MORI-IO and NIXL numbers are directly comparable. This is what tools/run_internode_io_benchmark.sh runs by default and what CI reports. Start here.

• Python (tests/python/io/benchmark.py) — kept for parity. Uses torchrun for rendezvous; convenient when you are already in a torch/torchrun workflow. Select it with run_internode_io_benchmark.sh --engine python.

Both accept the same workload flags (op, sweep, batch, QP, session); the C++ tool also accepts the Python spellings as aliases. See Differences from the Python flags.

The two-node runner picks the engine with --engine:

# C++ engine (default)
tools/run_internode_io_benchmark.sh --rank 0 --master-addr <ip> --ifname <nic> \
    -- --op write --all --enable-sess --enable-batch-transfer

# Python engine
tools/run_internode_io_benchmark.sh --engine python --rank 0 --master-addr <ip> \
    --ifname <nic> -- --op-type write --all --enable-sess --enable-batch-transfer

C++ Benchmark (default, nixlbench-matching)

The C++ benchmark, tests/cpp/io/bench_engine.cpp, is the default. Its measurement loop is written to match nixlbench (NVIDIA NIXL’s xferbench) exactly, so MORI-IO and NIXL RDMA numbers are apples-to-apples on the same fabric. Use it when you want a head-to-head comparison against NIXL, or a benchmark with no Python-interpreter overhead in the measurement loop (which the Python bench pays per iteration, so it shows up most at small message sizes).

What “nixl-matching” means here:

• One whole-loop timer around the entire iteration loop (not a per-iteration sum), identical to nixlbench’s total_timer.

• Latency is reported per single transfer: total_us / (iters × batch), matching nixl’s avg_latency = total_duration / (per_thread_iter × batch_size).

• Bandwidth = msg × batch × iters / 1e9 / (total_us / 1e6) (GB = 10⁹), same units as nixl.

• Completion is always an inline spin-poll in the benchmark thread (mirroring nixl’s getXferStatus scan); there is no cv-blocking wait that would add wakeup latency and make numbers non-comparable.

• Strict stop-and-wait (one request outstanding at a time, == nixl --pipeline_depth 1): post one request, spin until it completes, then post the next.

• Warmup iterations are excluded from timing (--warmup-iters, nixl --warmup_iter).

Build

bench_engine builds with the C++ tests (both options default ON):

cd /path/to/mori
cmake -B build -DBUILD_IO=ON -DBUILD_TESTS=ON
cmake --build build --target bench_engine -j
# binary: build/tests/cpp/bench_engine

bench_engine --help prints the full flag list, grouped by rendezvous, workload, sweep, memory, and backend tuning.

The MORI shared libraries must be discoverable at runtime — from the build tree above, or from python/mori after an editable install:

export LD_LIBRARY_PATH=$PWD/build/src/io:$PWD/build/src/application:$LD_LIBRARY_PATH
# or, after an editable install:
export LD_LIBRARY_PATH=/path/to/mori/python/mori:$LD_LIBRARY_PATH

Run (two nodes)

Unlike the Python benchmark (which uses torchrun for the out-of-band rendezvous), the C++ benchmark brings the two processes together over MORI’s own socket bootstrap — the same layer that starts multi-node CCO and SHMEM jobs, so connect and accept are retried over the MORI_BOOTSTRAP_TIMEOUT budget (seconds, default 300). Rank 0 is the initiator and drives every transfer; rank 1 is the target and only keeps its memory registered. Start the target first as a habit — rank 0 is the rendezvous root, so either order works and the two retry at each other, but the second process has to arrive within the budget. --master-ip is rank 0’s IP and is passed identically on both ranks; --self-ip is the address each rank advertises to its peer.

# On the TARGET node (rank 1) — start this FIRST:
MORI_DISABLE_AUTO_XGMI=1 build/tests/cpp/bench_engine \
    --rank 1 --master-ip 10.190.162.0 --self-ip 10.190.162.1 --port 18515 \
    --op write --all --sweep-start 1048576 --sweep-max 33554432 --sweep-step 1048576 \
    --iters 500 --warmup-iters 50 --num-qp-per-transfer 4 --enable-sess

# On the INITIATOR node (rank 0) — start this SECOND (prints the results table):
MORI_DISABLE_AUTO_XGMI=1 build/tests/cpp/bench_engine \
    --rank 0 --master-ip 10.190.162.0 --self-ip 10.190.162.0 --port 18515 \
    --op write --all --sweep-start 1048576 --sweep-max 33554432 --sweep-step 1048576 \
    --iters 500 --warmup-iters 50 --num-qp-per-transfer 4 --enable-sess

Both ranks must be given the same benchmark args (op, sweep, batch, etc.). Only rank 0 (initiator) prints the results table:

MsgSize(B)  Batch  Iters  AvgBW(GB/s)  AvgLat(us)  TotalDur(us)

--port is not the only TCP port in play, which matters behind a firewall:

Endpoint	Bound by	Address
Bootstrap rendezvous	rank 0 only	--master-ip : --port (18515 above)
Engine control plane	both ranks	--self-ip : --port + 1 + rank (18516 / 18517 above)
Bootstrap ring / allgather	both ranks	chosen interface, kernel-assigned ephemeral port

The engine control plane is a plain TCP channel: the initiator dials the target’s advertised EngineDesc address to run the QP handshake and to look up the remote memory region. No RDMA payload crosses it.

Sweep modes

• --all — sweep message size. Geometric ×2 by default, or linear when --sweep-step > 0, from --sweep-start to --sweep-max; batch stays at --transfer-batch-size.

• --all-batch — fix message size at --buffer-size and sweep batch 1,2,4,…,32768.

• neither — a single (--buffer-size, --transfer-batch-size) point.

Differences from the Python flags

Most flags share names with the Python benchmark (--op-type/--op, --num-qp-per-transfer, --num-worker-threads, --transfer-batch-size, --enable-sess, --batch-contiguous, --disable-chunking, --chunk-bytes, --poll_cq_mode, --mem-type, --iters, --sweep-step, --all, --all-batch), but note:

Python	C++ (nixl-style)	Notes
torchrun ... --node_rank	--rank 0 / --rank 1	C++ rendezvous runs over MORI’s socket bootstrap; rank 0 is the root, and the second process must arrive within MORI_BOOTSTRAP_TIMEOUT.
--host	--master-ip + --self-ip	--host is accepted as a spelling of --master-ip. --master-ip = rank 0’s IP, same on both ranks; --self-ip = the control-plane IP each rank advertises to the peer, which Python does not need because torchrun supplies it.
--src-gpu / --dst-gpu	--gpu / --target-dev-offset	Python’s names are accepted: --src-gpu is --gpu, and --dst-gpu N sets --target-dev-offset to N - gpu (passing both a conflicting --dst-gpu and --target-dev-offset is rejected). Under --xgmi-single-process they are the two local GPUs directly.
--num-initiator-dev / --num-target-dev	same	One process per GPU on each side, forked before any HIP call; initiator local i pairs with target local i, driving GPU --gpu + i. The two counts must be equal, as Python asserts. Each pair prints its own [gpu N]-prefixed row and rank 0 adds an [AGGREGATE] line summing bandwidth across pairs — Python prints per-rank tables with no aggregate.
--enable-batch-transfer (default OFF)	--enable-batch-transfer / --disable-batch-transfer (C++ default ON)	Same meaning as Python: ON = one N-descriptor batch request (the nixl-equivalent; nixl always batches); OFF = --transfer-batch-size N individual single-transfer submissions per iteration (Python run_single_once). The C++ tool defaults ON so the out-of-the-box run and --all-batch sweep are nixl-comparable; pass --disable-batch-transfer for the Python single-submission path.
--iters (default 128)	--iters (default 500), plus --warmup-iters	Only the warmup flag is new: it runs untimed iterations before the timed region, matching nixl’s --warmup_iter.
--sweep-start-size / --sweep-max-size	--sweep-start / --sweep-max (long names also accepted)	Names only; --sweep-step (0 = geometric ×2, >0 = linear) behaves the same in both.
--backend rdma|xgmi|fabric	same	--num-streams / --num-events size the XGMI and FABRIC HIP pools, as in Python. fabric (UALink scale-up, GPU memory only) needs a host whose GPUs expose /sys/bus/pci/devices/*/ualink; on RoCE-only hardware the engine reports No available backend found at the first transfer.
--xgmi-multiprocess (default OFF)	--xgmi-single-process (default OFF, i.e. multiprocess)	The defaults are opposite. The C++ tool’s two-rank path is already Python’s --xgmi-multiprocess: two processes with distinct engine keys, so the backend takes its hipIpc* route. --xgmi-single-process adds Python’s default instead — one process owning both GPUs, no rendezvous and no IPC. --xgmi-multiprocess is accepted to document intent but is a no-op.
(always on)	--skip-validate	Both tools verify transferred data as part of the run. Python compares the target’s buffer byte-for-byte over gloo; the C++ tool seeds a rank- and offset-dependent pattern and exchanges one checksum per transferred slot after the sweep (so the check is O(batch) on the wire but still covers every byte), printing validation: OK or naming the first mismatching slot and exiting non-zero. --skip-validate opts out; passing it to only one rank is safe — that rank contributes nothing and the run reports validation: skipped rather than hanging.

--self-ip must be an address the peer can reach over TCP, since it becomes the EngineDesc host the peer dials for the QP handshake and remote-MR lookup. 0.0.0.0 does not work: the peer ends up dialing its own loopback and fails with connect(...): Connection refused. It does not choose the RDMA device that carries the data — that stays rail/NUMA-aware (override with MORI_RDMA_DEVICES) — so the control IP need not be the RoCE interface’s. The bootstrap picks its local interface independently of both, so set MORI_SOCKET_IFNAME when that choice has no route to the peer.

MORI_DISABLE_AUTO_XGMI gates only the RDMA→XGMI fallback for hosts with no active RDMA device, and that fallback is off unless the variable is literally 0. The =1 above is therefore belt-and-braces, worth keeping only where something in the environment sets it to 0.

For --backend xgmi, run both ranks on the same host with different --gpu (XGMI is intra-node) and the same --master-ip/--self-ip; their engine control ports still differ because those are derived from the rank. The backend is created explicitly, so no env var is needed. That two-rank form measures the cross-process XGMI path (distinct engine keys ⇒ hipIpc*). To measure what the Python benchmark measures by default, pass --xgmi-single-process with --src-gpu/--dst-gpu: one process owns both GPUs, there is no rendezvous and no IPC, so --rank/--master-ip are not needed and small-message numbers become directly comparable to Python’s.

For multi-GPU runs (--num-initiator-dev N --num-target-dev N) the bootstrap world is 2N ranks, laid out initiators first. Engine control ports are derived from the global rank, so --port P reserves P for the bootstrap and P+1 … P+2N for the engines — leave that range free. Each side forks N-1 children before touching HIP, so a failure in one pair still reaps the others rather than leaving them holding ports.

Python Benchmark

Kept for parity with the C++ default (select it in the runner with --engine python). It uses torchrun for the out-of-band rendezvous instead of MORI’s socket bootstrap.

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH
export GLOO_SOCKET_IFNAME=ens14np0  # Set to your NIC interface

# Run on two nodes (replace node_rank and master_addr)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 \
    --master_addr="10.194.129.65" --master_port=1234 \
    tests/python/io/benchmark.py --host="10.194.129.65"

Fabric (cross-node scale-up UALink super-node)

The fabric backend transfers between two nodes that share the same scale-up fabric domain (same vPOD — verify with amd-smi fabric that PPOD_ID and VPOD_ID match and ACCEL_STATE is READY on both). Buffers are allocated as fabric-exportable VMM memory internally, so no --host/QP options are needed; OOB metadata is still exchanged over gloo (torchrun).

# node A (initiator)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 \
    --master_addr="<nodeA_ip>" --master_port=1234 \
    tests/python/io/benchmark.py --backend fabric \
    --all --enable-sess --enable-batch-transfer --transfer-batch-size 1 \
    --sweep-start-size 1048576 --sweep-max-size 268435456 --op-type read

# node B (target): same command with --node_rank=1

Requires ROCm ≥ 7.15 (HIP fabric VMM APIs). Only GPU memory is supported (--mem-type gpu). If the two nodes are not in the same vPOD, session creation fails fast with a clear error. The C++ benchmark also supports --backend fabric.

Benchmark Arguments

Argument	Description
--backend	rdma (cross-node), xgmi (intra-node), or fabric (cross-node UALink super-node, same vPOD)
--num-streams / --num-events	HIP stream/event pool size for xgmi/fabric backends (default 64)
--buffer-size	Message size per transfer (bytes)
--all	Sweep message size from 8B to 1MB
--sweep-start-size	Starting message size when using --all sweep
--sweep-max-size	Maximum message size when using --all sweep
--all-batch	Sweep batch size from 1 to 32768
--transfer-batch-size	Number of consecutive transfers
--enable-batch-transfer	Enable batch transfer mode
--enable-sess	Enable session transfer (lower latency)
--num-initiator-dev	Number of initiator devices
--num-target-dev	Number of target devices
--num-qp-per-transfer	Number of queue pairs used (default 4)
--op-type	Operation type: read or write
--poll_cq_mode	CQ polling mode: polling or event
--num-worker-threads	Number of worker threads (default 1; ignored when chunking/multi-NIC is on — posting is single-thread inline)
--disable-chunking	Disable single-transfer chunking (chunking is on by default)
--chunk-bytes	Chunk size in bytes when chunking is on (default 65536 = 64 KB)
--max-chunks	Max chunks per transfer (default 64)
--log-level	Log level (e.g., info)

Multi-NIC striping is controlled by the env var MORI_IO_NUM_NICS_PER_TRANSFER (default 1). For host memory, set it to the number of NUMA-local NICs and keep --num-qp-per-transfer ≥ 2 × NICs so each NIC gets ≥2 QPs. GPU memory should stay single-NIC (PCIe-bound).

Results: Thor2 RDMA Read

Config: 8 initiator + 8 target devices, session enabled, batch transfer, 2 QPs per transfer

torchrun --nnodes=2 --node_rank=1 --nproc_per_node=1 \
    --master_addr="10.235.192.57" --master_port=1234 \
    tests/python/io/benchmark.py --host="10.235.192.60" \
    --all --enable-sess --enable-batch-transfer \
    --num-qp-per-transfer 2 --num-target-dev 8 --num-initiator-dev 8

+--------------------------------------------------------------------------------------------+
|                                      Initiator Rank 7                                      |
+-------------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+----------------+---------------+---------------+--------------+--------------+
|      8      |      0.00      |      0.02     |      0.02     |    113.73    |    120.65    |
|      16     |      0.00      |      0.04     |      0.03     |    113.96    |    118.57    |
|      32     |      0.01      |      0.07     |      0.07     |    113.49    |    118.69    |
|      64     |      0.02      |      0.14     |      0.14     |    114.44    |    118.73    |
|     128     |      0.03      |      0.29     |      0.27     |    114.68    |    119.34    |
|     256     |      0.07      |      0.57     |      0.55     |    114.44    |    118.67    |
|     512     |      0.13      |      1.15     |      1.11     |    113.49    |    118.18    |
|     1024    |      0.26      |      2.30     |      2.23     |    114.20    |    117.78    |
|     2048    |      0.52      |      4.47     |      4.31     |    117.30    |    121.52    |
|     4096    |      1.05      |      8.31     |      8.01     |    126.12    |    130.95    |
|     8192    |      2.10      |     14.19     |     13.77     |    147.82    |    152.34    |
|    16384    |      4.19      |     22.16     |     21.56     |    189.30    |    194.58    |
|    32768    |      8.39      |     30.44     |     29.84     |    275.61    |    281.09    |
|    65536    |     16.78      |     37.51     |     36.55     |    447.27    |    458.96    |
|    131072   |     33.55      |     42.93     |     41.65     |    781.54    |    805.60    |
|    262144   |     67.11      |     45.66     |     45.08     |   1469.85    |   1488.69    |
|    524288   |     134.22     |     46.99     |     46.81     |   2856.02    |   2867.27    |
|   1048576   |     268.44     |     47.88     |     47.75     |   5605.94    |   5622.15    |
+-------------+----------------+---------------+---------------+--------------+--------------+

Results: Thor2 RDMA Write

Message Size Sweep

Config: 1 initiator + 1 target device, session enabled, batch transfer (128), 4 QPs, polling mode

numactl --cpunodebind=0 --membind=0 --physcpubind=0-47,96-143 \
    torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 \
    --master_addr="10.235.192.60" --master_port=1234 \
    tests/python/io/benchmark.py --host="10.235.192.60" \
    --enable-batch-transfer --enable-sess --buffer-size 1024 \
    --transfer-batch-size 128 --num-initiator-dev 1 --num-target-dev 1 \
    --num-qp-per-transfer 4 --all --num-worker-threads 1 \
    --log-level info --op-type write --poll_cq_mode polling

+--------------------------------------------------------------------------------------------------------+
|                                            Initiator Rank 0                                            |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | BatchSize | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
|      8      |    128    |      0.00      |      0.03     |      0.03     |    33.38     |    36.33     |
|      16     |    128    |      0.00      |      0.06     |      0.06     |    34.09     |    36.35     |
|      32     |    128    |      0.00      |      0.12     |      0.11     |    34.57     |    36.33     |
|      64     |    128    |      0.01      |      0.24     |      0.23     |    33.62     |    36.33     |
|     128     |    128    |      0.02      |      0.49     |      0.45     |    33.62     |    36.49     |
|     256     |    128    |      0.03      |      0.94     |      0.89     |    34.81     |    36.99     |
|     512     |    128    |      0.07      |      1.86     |      1.77     |    35.29     |    37.01     |
|     1024    |    128    |      0.13      |      3.84     |      3.53     |    34.09     |    37.09     |
|     2048    |    128    |      0.26      |      7.33     |      6.96     |    35.76     |    37.65     |
|     4096    |    128    |      0.52      |     12.94     |     12.46     |    40.53     |    42.09     |
|     8192    |    128    |      1.05      |     20.75     |     20.12     |    50.54     |    52.11     |
|    16384    |    128    |      2.10      |     29.03     |     28.33     |    72.24     |    74.02     |
|    32768    |    128    |      4.19      |     36.50     |     35.91     |    114.92    |    116.81    |
|    65536    |    128    |      8.39      |     41.74     |     41.39     |    200.99    |    202.70    |
|    131072   |    128    |     16.78      |     45.14     |     44.85     |    371.69    |    374.10    |
|    262144   |    128    |     33.55      |     46.93     |     46.76     |    715.02    |    717.56    |
|    524288   |    128    |     67.11      |     47.94     |     47.81     |   1399.99    |   1403.64    |
|   1048576   |    128    |     134.22     |     48.44     |     48.32     |   2770.90    |   2777.76    |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+

Batch Size Sweep

Config: 1 initiator + 1 target device, 1024B messages, session enabled, 16 QPs, polling mode

numactl --cpunodebind=0 --membind=0 --physcpubind=0-47,96-143 \
    torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 \
    --master_addr="10.235.192.60" --master_port=1234 \
    tests/python/io/benchmark.py --host="10.235.192.60" \
    --enable-batch-transfer --enable-sess --buffer-size 1024 \
    --transfer-batch-size 128 --num-initiator-dev 1 --num-target-dev 1 \
    --num-qp-per-transfer 16 --all-batch --num-worker-threads 1 \
    --log-level info --op-type write --poll_cq_mode polling

+--------------------------------------------------------------------------------------------------------+
|                                            Initiator Rank 0                                            |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | BatchSize | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
|     1024    |     1     |      0.00      |      0.10     |      0.09     |    10.25     |    11.10     |
|     1024    |     2     |      0.00      |      0.19     |      0.18     |    10.73     |    11.33     |
|     1024    |     4     |      0.00      |      0.34     |      0.33     |    12.16     |    12.60     |
|     1024    |     8     |      0.01      |      0.58     |      0.54     |    14.07     |    15.07     |
|     1024    |     16    |      0.02      |      0.83     |      0.72     |    19.79     |    22.87     |
|     1024    |     32    |      0.03      |      1.48     |      1.30     |    22.17     |    25.19     |
|     1024    |     64    |      0.07      |      2.45     |      2.14     |    26.70     |    30.61     |
|     1024    |    128    |      0.13      |      3.39     |      3.06     |    38.62     |    42.87     |
|     1024    |    256    |      0.26      |      4.43     |      4.19     |    59.13     |    62.53     |
|     1024    |    512    |      0.52      |      5.08     |      4.84     |    103.24    |    108.37    |
|     1024    |    1024   |      1.05      |      5.79     |      5.43     |    181.20    |    192.99    |
|     1024    |    2048   |      2.10      |      6.47     |      6.24     |    324.01    |    336.00    |
|     1024    |    4096   |      4.19      |      7.00     |      6.85     |    599.15    |    611.94    |
|     1024    |    8192   |      8.39      |      7.15     |      6.76     |   1173.02    |   1241.83    |
|     1024    |   16384   |     16.78      |      7.29     |      7.02     |   2301.69    |   2390.62    |
|     1024    |   32768   |     33.55      |      7.32     |      7.27     |   4585.50    |   4617.83    |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+

Results: CX7 RDMA (Batch Size = 1)

Write

Config: 1 initiator + 1 target device, single transfer, session enabled, message sweep 1KB-64MB

torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 \
    --master_addr="10.194.132.29" --master_port=1234 \
    tests/python/io/benchmark.py --host="10.194.132.29" \
    --transfer-batch-size 1 --all --sweep-start-size=1024 \
    --sweep-max-size=67108864 --op-type write \
    --enable-sess --enable-batch-transfer

+--------------------------------------------------------------------------------------------------------+
|                                            Initiator Rank 0                                            |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | BatchSize | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
|     1024    |     1     |      0.00      |      0.20     |      0.17     |     5.25     |     5.87     |
|     2048    |     1     |      0.00      |      0.37     |      0.35     |     5.48     |     5.81     |
|     4096    |     1     |      0.00      |      0.75     |      0.70     |     5.48     |     5.88     |
|     8192    |     1     |      0.01      |      1.49     |      1.37     |     5.48     |     5.96     |
|    16384    |     1     |      0.02      |      2.86     |      2.68     |     5.72     |     6.11     |
|    32768    |     1     |      0.03      |      5.29     |      4.94     |     6.20     |     6.64     |
|    65536    |     1     |      0.07      |     10.18     |      9.11     |     6.44     |     7.20     |
|    131072   |     1     |      0.13      |     16.17     |     15.44     |     8.11     |     8.49     |
|    262144   |     1     |      0.26      |     24.43     |     23.77     |    10.73     |    11.03     |
|    524288   |     1     |      0.52      |     32.82     |     31.97     |    15.97     |    16.40     |
|   1048576   |     1     |      1.05      |     39.62     |     38.91     |    26.46     |    26.95     |
|   2097152   |     1     |      2.10      |     43.98     |     43.56     |    47.68     |    48.15     |
|   4194304   |     1     |      4.19      |     46.54     |     46.37     |    90.12     |    90.44     |
|   8388608   |     1     |      8.39      |     48.00     |     47.89     |    174.76    |    175.18    |
|   16777216  |     1     |     16.78      |     48.77     |     48.71     |    344.04    |    344.46    |
|   33554432  |     1     |     33.55      |     49.17     |     49.13     |    682.35    |    683.02    |
|   67108864  |     1     |     67.11      |     49.36     |     49.34     |   1359.46    |   1360.09    |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+

Read

Config: Same as write, but with --op-type read

torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 \
    --master_addr="10.194.132.29" --master_port=1234 \
    tests/python/io/benchmark.py --host="10.194.132.29" \
    --transfer-batch-size 1 --all --sweep-start-size=1024 \
    --sweep-max-size=67108864 --op-type read \
    --enable-sess --enable-batch-transfer

+--------------------------------------------------------------------------------------------------------+
|                                            Initiator Rank 0                                            |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | BatchSize | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
|     1024    |     1     |      0.00      |      0.17     |      0.15     |     5.96     |     6.74     |
|     2048    |     1     |      0.00      |      0.34     |      0.31     |     5.96     |     6.63     |
|     4096    |     1     |      0.00      |      0.69     |      0.63     |     5.96     |     6.49     |
|     8192    |     1     |      0.01      |      1.32     |      1.23     |     6.20     |     6.66     |
|    16384    |     1     |      0.02      |      2.55     |      2.35     |     6.44     |     6.96     |
|    32768    |     1     |      0.03      |      4.58     |      4.40     |     7.15     |     7.44     |
|    65536    |     1     |      0.07      |      8.33     |      7.96     |     7.87     |     8.23     |
|    131072   |     1     |      0.13      |     14.47     |     13.57     |     9.06     |     9.66     |
|    262144   |     1     |      0.26      |     21.56     |     20.97     |    12.16     |    12.50     |
|    524288   |     1     |      0.52      |     29.32     |     28.63     |    17.88     |    18.31     |
|   1048576   |     1     |      1.05      |     35.47     |     34.95     |    29.56     |    30.00     |
|   2097152   |     1     |      2.10      |     39.62     |     39.38     |    52.93     |    53.25     |
|   4194304   |     1     |      4.19      |     42.09     |     41.95     |    99.66     |    99.99     |
|   8388608   |     1     |      8.39      |     43.49     |     43.38     |    192.88    |    193.39    |
|   16777216  |     1     |     16.78      |     44.23     |     44.17     |    379.32    |    379.83    |
|   33554432  |     1     |     33.55      |     44.59     |     44.57     |    752.45    |    752.90    |
|   67108864  |     1     |     67.11      |     44.81     |     44.78     |   1497.51    |   1498.58    |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+

Benchmark Tuning Guide

This section is a practical starting point for maximizing single-NIC internode RDMA write bandwidth with large messages and batched transfers. Latency-bound or single-transfer workloads may prefer different settings.

Recommended starting config

These workload flags apply to both engines (the C++ default and Python); only the launch wrapper differs. Shown against the C++ binary:

tests/cpp/io/bench_engine --op write \
    --enable-sess --enable-batch-transfer --batch-contiguous \
    --disable-chunking \
    --num-qp-per-transfer 8 --num-worker-threads 8 \
    --transfer-batch-size 128 --num-initiator-dev 1 --num-target-dev 1
    # (plus --rank/--master-ip/--self-ip/--port for the two-node rendezvous)

On a 400G-class NIC this typically reaches ~95%+ of link rate across the 1 MiB–32 MiB message range and removes the mid-size bandwidth dip seen with the defaults. Tune from here.

Parameters that help

Knob	Effect
--enable-batch-transfer --enable-sess	Submit the whole batch through a persistent session, cutting per-transfer overhead.
--batch-contiguous	Contiguous per-transfer offsets let consecutive transfers merge into fewer, larger RDMA work requests (WRs). This is usually the biggest single win at large sizes and flattens the message-size curve. Do not use it when you intend to stress the send queue / keep one WR per transfer.
--num-qp-per-transfer	More QPs pipeline a large transfer across the NIC. Bandwidth generally scales up to ~8 QPs and then plateaus; 16+ gives no further gain. Start at 8.
--disable-chunking	Chunking splits each transfer into --chunk-bytes pieces; for large messages this multiplies WRs and can saturate the send queue, collapsing bandwidth at the largest sizes. Turning it off avoids that — but read the WR-size limit below.
--num-worker-threads	With chunking off + --batch-contiguous, use several workers (e.g., 8). Besides parallel posting, this splits a large batch across workers so each merged WR stays under the NIC message-size limit. With chunking on, this knob has no bandwidth effect (posting is single-thread inline).

The merged-WR size limit

A single RDMA WR cannot exceed the NIC port’s reported max_msg_sz (ibv_port_attr, 2 GiB on the usual mlx5 and bnxt_re parts), taken as the minimum across the endpoints a transfer uses. With --batch-contiguous --disable-chunking and a single worker, an entire batch can merge into one WR of message_size × transfer_batch_size. If that exceeds max_msg_sz the run aborts with:

merged RDMA WR ... exceeds local max_msg_sz ...; enable RDMA transfer chunking or reduce batch size

Any one of these avoids it:

• Use multiple --num-worker-threads (splits the batch so each merged WR is smaller) — recommended.

• Keep message_size × transfer_batch_size ≤ max_msg_sz.

• Re-enable chunking (drop --disable-chunking) so oversized transfers are split automatically.

Parameters that usually do not help (or can hurt)

• MORI_IO_QP_MAX_SEND_WR / --max-send-wr: no benefit once chunking is off (merged WRs are few); leave at default.

• --max-msg-sge: raising it can reduce bandwidth; leave at default.

• MORI_RDMA_DEVICES (manual NIC pinning): the backend’s automatic NIC selection is rail/NUMA-aware. Overriding it can pin traffic to a non-local or cross-rail NIC and sharply reduce bandwidth. Only override if you have verified the topology and the auto-choice is wrong.

• Lossless traffic class / service level (MORI_IO_TC / MORI_IO_SL): only helps if the fabric is actually configured for PFC on that priority. On a fabric that is not, setting these can sharply reduce bandwidth. Leave unset unless PFC is configured end-to-end.

Host (CPU) vs GPU memory

• GPU device memory (registered via dmabuf) is unaffected by the notes below.

• For host (CPU) memory, on NICs with a small memory-translation (PTE) cache, a large 4 KiB-paged buffer used as the RDMA-write source can lose bandwidth past a few tens of MiB per transfer, and very large regions may fail memory registration outright. Back host buffers with hugepages (explicit hugetlb or THP) in that case. The read direction and the passive (target) side are less sensitive.

• Multi-NIC striping (MORI_IO_NUM_NICS_PER_TRANSFER) is intended for host memory; keep GPU memory on a single NIC (it is PCIe-bound).