GEMM + reduce-scatter
GemmReduceScatterOp is the GEMM + all-reduce pipeline
stopped before the gather, for sequence parallelism. Each rank multiplies its
activations by its shard of a row-parallel weight; rank r keeps rows r*M/P
onward of the sum: the dim-0 split reduce_scatter_tensor makes, and what a
sequence-parallel layer keeps after a RowParallelLinear. The GEMM’s epilogue
pushes each destination’s row band as it is produced, so the transfer overlaps
the rest of the GEMM, and the reduce then sums this rank’s band. It lives in the
same module, mori.ops.gemm_ar, and subclasses GemmAllReduceOp.
Table of Contents
Quick Reference
from mori.ops.gemm_ar import GemmReduceScatterOp, preshuffle_a_scale, preshuffle_b
# comm: a mori.cco.Communicator whose per_rank_vmm was sized from
# GemmReduceScatterOp.window_bytes_for(world, **settings); see section 10.
settings = dict(n=5120, k=2048, m_max=16384, quant="mxfp8") # V4.1-Flash wo_b, TP4
with GemmReduceScatterOp(comm, **settings) as op:
op.self_test() # collective, once per op
b_shuffled = preshuffle_b(b_fp8) # once per weight
b_scale = w_exps.t().contiguous().to(torch.int32).reshape(-1) # [N/32, K/32] ue8m0
m_pad = op.padded_m(x.shape[0])
a_fp8, a_exps = quantize_mxfp8(op.pad_rows(x, m_pad)) # your quantiser
band = op(a_fp8, b_shuffled, preshuffle_a_scale(a_exps), b_scale).clone() # [m_pad / P, N] bf16
torch.cuda.synchronize() # before close() frees the window
What |
Import |
|---|---|
The operator |
|
Weight and mxfp8 A-scale layouts |
|
Communicator |
|
The all-reduce it extends |
|
1. What it computes
Rank r of P passes its activations A_r and its weight shard B_r, and
receives its row band of the sum:
band [M/P, N] bf16 = rows r*M/P .. (r+1)*M/P - 1 of
sum over ranks of dequant(A_r [M, K]) @ dequant(B_r [N, K])^T
As for the all-reduce, K is the per-rank K and N is not sharded; the GEMM
accumulates in fp32, the partials travel as bf16, and the reduce adds them in
fp32 and rounds once to bf16.
Shape |
dtype |
|
|---|---|---|
A ( |
|
|
B ( |
|
|
Scales |
per |
fp32, int32, uint8, or none |
Result |
|
bf16 |
The padding decides the bands. The split is over the padded M, and the
padded rows, which come back as zeros, sit at the end, in the last band or
bands: rank r holds max(0, min(M/P, rows - r*M/P)) real rows. A
sequence-parallel layer that shards the unpadded token count would disagree
about which rows each rank holds, so pad the sequence to op.padded_m() before
it is sharded.
The layer is a RowParallelLinear under sequence parallelism; the measured one is
DeepSeek-V4.1-Flash’s wo_b at TP4 (N=5120, K=2048, mxfp8).
2. How it works
No kernel differs from the all-reduce’s (how that works). The GEMM pushes each destination’s band in chunks from its epilogue, with the same default chunk rule (the largest divisor of the bands per destination, up to 8). Then three kernels run:
Drain: waits for this rank’s pushes to land, then a cross-rank barrier.
Reduce: sums the
Pcontributions to this rank’s band.rs_barrier: a barrier-only kernel standing in for the one the gather used to provide.
The last barrier is load-bearing. The reduce reads receive slots that the peers
fill, and the next call’s epilogue refills them, so without it a rank one call
ahead overwrites a slower peer’s slot mid-reduce. Before it existed this showed
as relL2 0.71 on one call of an alternating-M test; the test now holds one rank
between drain and reduce, and runs the same schedule without rs_barrier as a
negative control that has to corrupt.
The window is the all-reduce’s: it still holds a whole [m_max, N] output
region, of which this op writes 1/P, so it is about 700 MiB per rank at
m_max=16384, N=7168, P=8 and 520 MiB at N=5120, P=4. Shrinking it would
change the layout every other region is placed by.
3. Requirements
The same as for GEMM + all-reduce:
MI355X (gfx950), and the FlyDSL extra (
pip install 'amd_mori[flydsl]').MORI built with
BUILD_CCO_SDMA=ON(the default) andMORI_ENABLE_SDMA=1at run time. The constructor does not check that the communicator has SDMA queues, soself_test()is the only guard.One node,
P= 2, 4 or 8; every rank constructs the op and makes every call.A
Communicatorwhoseper_rank_vmmis sized fromGemmReduceScatterOp.window_bytes_for, called with the constructor’s settings.
The rules shared by every fused operator are in What every operator has in common.
4. Constructor
GemmReduceScatterOp(comm, *, n, k, m_max, block_m=None, block_n=256,
sdma_queues=1, quant="blockscale", max_shapes=None)
Name |
Type |
Default |
Description |
|---|---|---|---|
|
|
(required) |
The communicator; |
|
|
(required) |
Output width N, a multiple of 256. There is no gather, so no fp8-wire rule. |
|
|
(required) |
Per-rank K. fp8 quants: a multiple of 128, at least 256. |
|
|
(required) |
Largest per-rank M before the split (the whole sequence), at least 1. Rounded up to a multiple of |
|
|
|
Tile rows and padding granule. |
|
|
|
Tile columns; only 256 is accepted. |
|
|
|
SDMA queues per (source, destination) pair, at least 1. One is all the pipeline uses, and the measurements below use 1. |
|
|
|
|
|
|
|
Most distinct M the instance serves; |
Not accepted. The all-reduce’s gather_dtype, gather_transport,
scatter_dtype and schedule are not parameters, so passing one raises
TypeError. The scatter is the only leg and stays bf16, so there is no fp8
wire, and M pads by the default rule, so the bands are even: there is no
schedule="wo_b". The attributes keep the all-reduce’s defaults ("bf16",
"lsa", "bf16", "default"). Construction is collective, raises
ValueError on a rule from section 7 before allocating
anything, and releases what it acquired if the communicator fails.
window_bytes_for()
GemmReduceScatterOp.window_bytes_for(world_size, *, m_max, n, block_m=None,
max_shapes=None, k=None, quant="blockscale") -> int
Static. Returns GemmAllReduceOp.window_bytes_for for these arguments with the
default wires and schedule, which is what the constructor allocates. Pass the
constructor’s quant, and block_m if it set one: when block_m is None the
tile, and so the padding, follows the quant. k is optional. Raises
ValueError for an unknown quant, a block_m the quant forbids, a
non-positive size, or a world size the window cannot take.
5. Calling the operator
band = op(a_fp8, b_preshuffled, a_scale=None, b_scale=None)
The operands are exactly the all-reduce’s, including the scale layouts for each
quant (section 5 there); for
quant="bf16" they are bf16 A and the checkpoint’s row-major [N, K] bf16
weight, with no scales.
Argument |
Shape |
dtype |
Notes |
|---|---|---|---|
|
|
|
The whole padded sequence: M a multiple of |
|
|
|
|
|
per |
fp32, int32 or uint8 |
|
|
per |
fp32, int32 or uint8 |
|
There is no logical_m: M itself must be padded, and passing it raises
TypeError.
Returns an [M/P, N] bf16 row-major tensor: a view of this rank’s band in
the op’s window, rows rank*M/P onward of the summed GEMM, valid once the work
queued on the current stream has run. This rank’s next call overwrites it and
close() frees it: order its readers before the next call (same stream, or an
event the next call’s stream waits on), and .clone() it to keep it. The first
call at a new M compiles that M’s kernels; one instance is not usable
concurrently.
6. Other methods
All inherited from GemmAllReduceOp; the
all-reduce page has the details.
padded_m()
op.padded_m(m) -> int: m rounded up to a multiple of P * block_m, which is
also the granule a sequence-parallel caller must pad to before sharding.
ValueError if m < 1.
pad_rows()
op.pad_rows(x, m_pad) -> Tensor: zero-extends a 2-D [M, K] tensor to
m_pad rows in a buffer the op reuses, which the next pad_rows overwrites.
Pad before quantising, so the scales cover the padded rows.
self_test()
op.self_test(m=None), collective: writes rank + 1 over the input region,
runs scatter, reduce and rs_barrier without a GEMM, and checks that every
element of this rank’s band is P * (P + 1) / 2 within 1%; otherwise it raises
RuntimeError: gemm_ar self-test failed: the reduce-scatter returned .... m
defaults to m_max. Call it once on every rank after construction, before any
capture.
close() and the context manager
op.close() deregisters the window and frees its memory; it is idempotent, and
a closed op raises RuntimeError. Synchronise first if work may be in flight.
with GemmReduceScatterOp(...) as op: calls close() on exit.
Attributes
window_bytes (equal to window_bytes_for), m_max (padded), the
constructor’s settings and the inherited wire settings are plain attributes,
for reading; dev_comm.sdma_num_queue is the number of SDMA queues per peer the
communicator actually connected.
7. Shapes and limits
The rules are the all-reduce’s (its table) without the wire options. The ones that differ or matter most here:
Constraint |
Value |
Error if violated |
|---|---|---|
Call M |
a multiple of |
|
Call M |
at most |
|
All-reduce-only options |
not accepted |
|
|
as for the all-reduce |
|
K |
fp8 quants: a multiple of 128, at least 256; |
|
N |
a multiple of 256 |
|
Ranks |
2, 4 or 8 |
|
Operands and scales |
as for the all-reduce |
|
Lifetime |
not closed |
|
Window |
fits the free VMM reservation |
|
8. Choosing settings
Setting |
What the default picks |
Change it when |
|---|---|---|
|
|
the checkpoint is mxfp8 ( |
|
128 for blockscale, 256 otherwise |
rarely: blockscale takes other granule-legal tiles, and bf16 also takes 128, which doubles the grid at small M; the mxfp8 forms need 256 |
Chunks |
the largest divisor of the bands per destination, up to 8: at V4.1-Flash TP4 that is 2 at M=2048, 4 at 4096 and 8 above |
not a parameter |
Wires |
bf16 scatter, no gather |
not available |
|
1 |
keep |
|
every padded M up to |
keep |
Whether to fuse at your M. At V4.1-Flash TP4 the fused op was ahead of the
GEMM followed by reduce_scatter_tensor at every M measured, 2048 included. At
M=2048 there are only two chunks per destination and it trails the split SDMA
path by 2.5%, but that split has no public op.
The operator reads no environment variable of its own. MORI_ENABLE_SDMA=1 is
required, MORI_SOCKET_IFNAME picks the bootstrap interface (the tests use
lo), and MORI_GEMM_AR_PULL_BLOCKS has no effect because there is no gather.
9. Performance
DeepSeek-V4.1-Flash wo_b ([M, 5120], K=2048, mxfp8) on 4x MI355X (ROCm 7.2,
FlyDSL 0.4.0), from the
README:
bench_gemm_ar.py --collective rs --sdma-queues 1, graph replay, maximum over
ranks, median of 3 interleaved rounds on an idle box (spread at most 4.7% at
M=2048 and 2.7% above). The baseline is the GEMM followed by
reduce_scatter_tensor, what a serving stack runs today. µs; the fused column
gives the chunks per destination:
M |
GEMM only |
GEMM + |
split SDMA |
fused |
vs RCCL |
vs split SDMA |
|---|---|---|---|---|---|---|
2048 |
39.6 |
161.7 |
152.8 |
156.6 (2) |
-3.2% |
+2.5% |
4096 |
66.4 |
295.4 |
269.9 |
262.9 (4) |
-11.0% |
-2.6% |
8192 |
104.4 |
540.5 |
484.2 |
460.2 (8) |
-14.9% |
-5.0% |
16384 |
174.9 |
1065.3 |
917.3 |
834.5 (8) |
-21.7% |
-9.0% |
Two separate effects. The copy engines alone beat RCCL by 5-14% (split SDMA against GEMM +
reduce_scatter_tensor); fusing then hides about 83 of the 175 µs of GEMM at M=16384.At M=2048 fusing loses to split SDMA, which has no public op; against the split a caller actually has, the fused op is ahead at every M measured.
The link is still the limit: the unfused transfer is about 740 µs at M=16384, about 170 GB/s per rank over TP4’s three links.
The RCCL path validates at relL2 3.15e-3 against 2.37e-3 for the others. That is not an error: RCCL adds the partials in bf16 hop by hop, where the SDMA reduce accumulates in fp32 and rounds once.
10. Complete example
Run with MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=4 example.py.
quantize_mxfp8 stands for your quantiser; MORI ships none. It must return
a_fp8 as [m_pad, K] float8_e4m3fn and a_exps as the [m_pad, K/32]
ue8m0 exponent bytes, which preshuffle_a_scale turns into the flat int32
layout quant="mxfp8" reads (with quant="mxfp8_raw", pass them as uint8
instead).
import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ar import GemmReduceScatterOp, preshuffle_a_scale, preshuffle_b
# DeepSeek-V4.1-Flash wo_b at TP4; the same settings size the window and build the op.
SETTINGS = dict(n=5120, k=2048, m_max=16384, quant="mxfp8")
N, K = SETTINGS["n"], SETTINGS["k"]
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo") # only carries the unique id
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
vmm = 2 * GemmReduceScatterOp.window_bytes_for(world, **SETTINGS) + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with GemmReduceScatterOp(comm, **SETTINGS) as op:
op.self_test() # every rank, once, before any capture
# This rank's weight shard and its [N/32, K/32] ue8m0 scales; stand-ins.
w_fp8 = (torch.randn(N, K, device="cuda") / 8).to(torch.float8_e4m3fn)
w_exps = torch.full((N // 32, K // 32), 120, device="cuda", dtype=torch.uint8)
b = preshuffle_b(w_fp8)
b_scale = w_exps.t().contiguous().to(torch.int32).reshape(-1) # K-block major
def wo_b(x): # x: the whole sequence, [rows, K] bf16, rows <= m_max
a_fp8, a_exps = quantize_mxfp8(op.pad_rows(x, op.padded_m(x.shape[0])))
return op(a_fp8, b, preshuffle_a_scale(a_exps), b_scale) # [m_pad / P, N]
# Warm every M you will run (the first call compiles it), largest first.
for rows in (16384, 5000):
band = wo_b(torch.randn(rows, K, device="cuda", dtype=torch.bfloat16))
band_rows = op.padded_m(5000) // world
real = max(0, min(band_rows, 5000 - rank * band_rows)) # padding sits last
mine = band[:real].clone()
# CUDA graph: a static input, warm-up on a side stream, then capture.
x_static = torch.randn(16384, K, device="cuda", dtype=torch.bfloat16)
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
for _ in range(3):
wo_b(x_static)
torch.cuda.current_stream().wait_stream(side)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
band_static = wo_b(x_static) # a view of this rank's band in the window
x_static.copy_(torch.randn_like(x_static)) # new data, same buffer
graph.replay() # on every rank, in the same order
result = band_static.clone()
torch.cuda.synchronize() # nothing in flight when close() frees the window
dist.destroy_process_group()
11. Testing and benchmarking
# The public op's reduce-scatter cases (blockscale and bf16), on 2 ranks
BUILD_CCO_SDMA=ON MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
pytest tests/python/cco/test_gemm_ar_op.py -k reduce_scatter
# The section 9 table at M=16384; repeat per --mode and -m
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
torchrun --standalone --nproc_per_node=4 benchmark/cco/flydsl/gemm_ar/bench_gemm_ar.py \
--collective rs --sdma-queues 1 --mode fused-sdma --quant mxfp8 -m 16384 -n 5120 -k 2048
-k reduce_scatter selects test_reduce_scatter_returns_my_band_of_the_sum
(each rank’s band, its shape, two M through one instance, fresh operands, and the
self-test’s teeth) and test_reduce_scatter_survives_a_slow_reader (the
rs_barrier check and its negative control). The module’s worker runs all its
2-rank cases in one spawn, so it needs two GPUs.
With --collective rs the table’s modes are gemm-only, split-rccl,
split-sdma and fused-sdma; the benchmark refuses split-lsa,
--gather-dtype fp8, --scatter-dtype fp8 and --fuse-reduce-push. It
drives the same kernels through the low-level builders, prints a RESULT_JSON
line with max_rank_time_us, rel_l2 and validated, and defaults to
--quant ptpc and --sdma-queues 8, so pin both. The
all-reduce page has the rest of
the suite and the sweeps.
12. Troubleshooting
Symptom |
What it means |
|---|---|
|
Pad the whole sequence with |
|
Those are all-reduce options. This op has no gather, a bf16 scatter and the default padding. |
The bands do not match the next layer’s shards |
The bands split the padded M, with the padding in the last band or bands. Shard the padded sequence, or slice each band to its real rows. |
|
The SDMA puts moved nothing: MORI built with |
|
As for the all-reduce. |
Further reading
The
gemm_arREADME, next to the code, has the full design, every measurement table and the negative results; its “Fused GEMM + reduce-scatter” section covers this op.GEMM + all-reduce: the pipeline this op stops early, with the operand, scale and padding details it shares.
MORI-FUSED overview: what the six operators share, and the headline numbers.