GEMM + all-reduce

GemmAllReduceOp (in mori.ops.gemm_ar) multiplies each rank’s activations by its shard of a row-parallel weight and all-reduces the product, so every rank gets the [M, N] sum in bf16. It serves a RowParallelLinear under tensor parallelism, such as DeepSeek-V4’s attention output projection wo_b. The GEMM writes its C straight into the operator’s window, and the copy engine pushes each destination’s slice as soon as its last tile is written, so the reduce-scatter half of the all-reduce overlaps the rest of the GEMM. The reduce and all-gather run after it. The same pipeline stopped before the gather is GEMM + reduce-scatter.

Table of Contents

Quick Reference

from mori.ops.gemm_ar import GemmAllReduceOp, preshuffle_b

# comm: a mori.cco.Communicator whose per_rank_vmm was sized from
# GemmAllReduceOp.window_bytes_for(world, **settings); see section 10.
settings = dict(n=7168, k=2048, m_max=16384, quant="blockscale")  # V4-Pro wo_b, TP8
with GemmAllReduceOp(comm, **settings) as op:
    op.self_test()                                       # collective, once per op
    b_shuffled = preshuffle_b(b_fp8)                     # [N, K] e4m3fn, once per weight
    m_pad = op.padded_m(x.shape[0])
    a_fp8, a_scale = quantize_per_1x128(op.pad_rows(x, m_pad))  # your quantiser
    out = op(a_fp8, b_shuffled, a_scale, b_scale)[: x.shape[0]].clone()  # [M, N] bf16
    torch.cuda.synchronize()                             # before close() frees the window

What

Import

The operator

from mori.ops.gemm_ar import GemmAllReduceOp

Weight and mxfp8 A-scale layouts

from mori.ops.gemm_ar import preshuffle_b, preshuffle_a_scale

Shape predicate

from mori.ops.gemm_ar import supports

Communicator

from mori.cco import Communicator, UniqueId

Reduce-scatter variant

from mori.ops.gemm_ar import GemmReduceScatterOp (its page)

GEMM and GEMV with nothing fused

from mori.ops.gemm_ar import Mxfp8GemmOp, Mxfp8GemvOp

1. What it computes

Rank r of P passes its activations A_r and its weight shard B_r, and every rank receives

out [M, N] bf16 = sum over r of  dequant(A_r [M, K]) @ dequant(B_r [N, K])^T

K is the per-rank K, the slice of the layer’s input this rank holds; N is not sharded. The GEMM applies the quant scales and accumulates in fp32, and the reduce adds the P partials in fp32 and rounds once to bf16.

Shape

dtype

A (a_fp8)

[M, K], M = op.padded_m(rows), the same on every rank

float8_e4m3fn; bf16 with quant="bf16"

B (b_preshuffled)

[N, K]

float8_e4m3fn through preshuffle_b; with quant="bf16" the row-major bf16 weight

Scales

per quant, see section 5

fp32, int32, uint8, or none

Result

[M, N] row-major; padded rows are zero

bf16

With bf16 wires every rank gets the same bytes; with gather_dtype="fp8" the bands a rank receives come through e4m3 and its own band does not. The layer is a RowParallelLinear under tensor parallelism: the measured ones are the wo_b of DeepSeek-V4-Pro (TP8) and DeepSeek-V4.1-Flash (TP4) and Kimi-K3’s MLP down_proj; quant="bf16" serves layers a checkpoint keeps in BF16, such as Kimi-K3’s attention o_proj.

2. How it works

A call issues four or more kernels on the current stream:

  1. GEMM + scatter. The GEMM writes C into the window. Its rows split into P bands, one per destination, and each band into chunks; when a chunk’s last tile is written, the copy engine pushes the chunk into the destination’s receive slot. Tiles are walked chunk by chunk with the destination rotated by rank, so every link starts early.

  2. Drain: waits for this rank’s pushes to land, then a cross-rank barrier.

  3. Reduce: sums the P contributions to this rank’s band.

  4. Gather: sends the reduced band to every peer. An fp8 gather adds a quantise kernel, then a CU pull that widens on the way in ("lsa") or a copy-engine push and a dequantise kernel ("sdma").

Only the scatter overlaps the GEMM, so the gain depends on how much GEMM there is to hide it behind. A destination’s band is M / (P * block_m) tile rows, which the default plan pushes in the largest number of chunks, up to 8, that divides that count: more chunks start the first push earlier until the copy engine’s per-packet cost shows, and 8 is the measured knee. At M = P * block_m there is one chunk, pushed after the GEMM, and nothing overlaps.

The window holds C for m_max rows, the [m_max, N] bf16 result, a tmp slot for this rank’s reduced slice (m_max / P rows), P receive slots of m_max / P rows, an fp8 copy of the result when the gather is fp8, and control words (barrier flags, a counter set per distinct M, submit locks), all laid out for m_max. It is about 700 MiB per rank at m_max=16384, N=7168, P=8 with bf16 wires, and 812 MiB with an fp8 gather.

3. Requirements

Requirement

Detail

GPU

MI355X (gfx950). The GEMM kernels are written for gfx950 only and every measurement is on MI355X. The constructor does not check the GPU; the only architecture check in the code is schedule="wo_b", which keeps the default plan anywhere else.

FlyDSL

pip install 'amd_mori[flydsl]'. import mori.ops.gemm_ar, preshuffle_b and preshuffle_a_scale work without it; touching GemmAllReduceOp raises ModuleNotFoundError.

SDMA

MORI built with BUILD_CCO_SDMA=ON (the default) and MORI_ENABLE_SDMA=1 at run time. The constructor does not check that the communicator has SDMA queues, so self_test() is the only guard.

Ranks

One node (one LSA domain), P = 2, 4 or 8. Every rank constructs the op and makes every call.

Memory

A Communicator whose per_rank_vmm is sized from window_bytes_for, called with the constructor’s settings. Each op allocates its own window from that reservation, so add up the windows of every op alive at once.

The rules this op shares with the other fused operators (padding, result aliasing, the collective contract, cleanup, fp8 operand and scale layouts) are in What every operator has in common.

4. Constructor

GemmAllReduceOp(comm, *, n, k, m_max, block_m=None, block_n=256, sdma_queues=1,
                gather_dtype="bf16", gather_transport="lsa", quant="blockscale",
                max_shapes=None, scatter_dtype="bf16", schedule="default")

Name

Type

Default

Description

comm

mori.cco.Communicator

(required)

The communicator; P = comm.nranks. The op does not take ownership.

n

int

(required)

Output width N: a multiple of 256, and of 1024 with gather_dtype="fp8".

k

int

(required)

Per-rank K. fp8 quants: a multiple of 128, at least 256. "bf16": a multiple of 64, at least 128.

m_max

int

(required)

Largest per-rank M the op will serve, at least 1. Rounded up to a multiple of P * block_m (op.m_max holds the rounded value). Sizes the window, which cannot grow.

block_m

int | None

None

Tile rows, which are also the padding granule. None picks the quant’s tile: 128 for "blockscale", 256 otherwise. "mxfp8" and "mxfp8_raw" accept only 256, "bf16" 128 or 256, and "blockscale" any positive multiple of 128, though 128 is its measured tile (256x256 spills registers).

block_n

int

256

Tile columns. Only 256 is accepted: the epilogue’s lane transpose pairs exactly two N-tiles.

sdma_queues

int

1

SDMA queues per (source, destination) pair, at least 1. One is all the pipeline uses; more only adds idle queues, and in a process that already holds SDMA queues their creation fails. The first dev-comm on a communicator fixes the count.

gather_dtype

str

"bf16"

Wire dtype of the all-gather leg: "bf16" (exact) or "fp8" (e4m3 with one fp32 scale per row, half the bytes; relL2 2.5e-2 against 2.4e-3 with a bf16 wire).

gather_transport

str

"lsa"

Who moves an fp8 gather: "lsa" (the CUs pull each peer’s band over xGMI and widen it, one kernel) or "sdma" (the copy engines push, then a dequantise kernel). No effect with a bf16 gather.

quant

str

"blockscale"

Operand contract. "blockscale": fp8, A 1x128 and B 128x128 fp32 scales (DeepSeek-V4-Pro). "mxfp8": fp8, 32-wide ue8m0 scales, A’s through preshuffle_a_scale (DeepSeek-V4.1-Flash). "mxfp8_raw": the same numbers with the quantiser’s row-major uint8 scales, staged through LDS. "bf16": bf16 A and B, no scales, bf16 scatter wire. There is no "ptpc".

max_shapes

int | None

None

Most distinct M plans the instance serves, at least 1; each takes a counter set. None reserves one for every padded M up to m_max, plus the wo_b plans, so a legal call never runs out.

scatter_dtype

str

"bf16"

Wire dtype of the scatter leg: "bf16" or "fp8", a lossy opt-in (the epilogue quantises with one fp32 scale per row and 256-column tile). Not with quant="bf16".

schedule

str

"default"

"default" or "wo_b", the measured MI355X padding and chunk profile for two wo_b targets (section 8). With "wo_b", pass logical_m= on every call.

Construction is collective. It raises ValueError for a setting outside these rules (section 7) before allocating anything, and RuntimeError from the communicator if the window does not fit; on failure it releases whatever it had acquired.

window_bytes_for()

GemmAllReduceOp.window_bytes_for(world_size, *, m_max, n, block_m=None,
    gather_dtype="bf16", gather_transport="lsa", max_shapes=None, k=None,
    quant="blockscale", scatter_dtype="bf16", schedule="default") -> int

Static. The bytes the window will take, by the constructor’s own planning rules: op.window_bytes equals it for the same settings. world_size is comm.nranks; the rest mean what they mean for the constructor, and k may be omitted except with schedule="wo_b". block_n and sdma_queues do not change the window. It raises ValueError for an unknown quant, a block_m the quant forbids, a non-positive size, or a world size or fp8-gather N the window cannot take; N and K against the GEMM are the constructor’s checks.

5. Calling the operator

out = op(a_fp8, b_preshuffled, a_scale=None, b_scale=None, *, logical_m=None)

Argument

Shape

dtype

Notes

a_fp8

[M, K]

float8_e4m3fn (bf16 with quant="bf16")

M = op.padded_m(rows), the same on every rank, at most op.m_max. On the GPU; a non-contiguous tensor is copied.

b_preshuffled

[N, K]

float8_e4m3fn (bf16 with quant="bf16")

preshuffle_b(w), once per weight (the byte order of aiter’s shuffle_weight(x, layout=(16, 16))); for "bf16" the checkpoint’s row-major weight, not shuffled.

a_scale, b_scale

per quant, below

per quant, below

On the GPU; None for "bf16".

logical_m

int

Keyword-only: the rows before padding. Selects the fitted plan with schedule="wo_b"; otherwise optional, and op.padded_m(logical_m) must equal M.

quant

a_scale

b_scale

"blockscale"

fp32, M*K/128 values: [M, K/128] column-major (stride (1, M)), [K/128, M], or flat with (row, kb) at kb * M + row

fp32 [N/128, K/128] row-major

"mxfp8"

int32, flat: preshuffle_a_scale(a_exps), a_exps the [M, K/32] ue8m0 bytes

int32, flat: w_exps.t().contiguous().to(torch.int32).reshape(-1) from the [N/32, K/32] bytes

"mxfp8_raw"

uint8 [M, K/32] as the quantiser emits it, 2-D or flat

uint8 [N/32, K/32], 2-D or flat

A blockscale a_scale that is [M, K/128] row-major is refused as ambiguous: pass .reshape(-1) for a K/128-major buffer (what aiter_per1x128_quant(transpose_scale=True) returns) or .t() for a logical row-major tensor.

Returns an [M, N] bf16 row-major tensor that is a view of the op’s window, holding the result once the work queued on the current stream has run. Every M’s result starts at the same address, so this rank’s next call overwrites it and close() frees it: order its readers before the next call (same stream, or an event the next call’s stream waits on), and .clone() it to keep it. Padded rows are zero; slice [:logical_m]. The first call at a new M compiles that M’s kernels. One instance is not usable concurrently: the window, the kernel cache and the padding buffer are shared.

6. Other methods

padded_m()

op.padded_m(m) -> int: the physical rows for m logical rows, m rounded up to a multiple of P * block_m or, with schedule="wo_b", the fitted plan’s rows at a profiled M (8200 gives 9216 at V4-Pro TP8, 4200 gives 4224 at V4.1-Flash TP4). It keeps no state. ValueError if m < 1. The module function padded_m(m, world_size, block_m=128) applies the default rule.

pad_rows()

op.pad_rows(x, m_pad) -> Tensor: zero-extends a 2-D [M, K] tensor to m_pad rows in a buffer the op reuses and returns a view of it, which the next pad_rows overwrites. Pad before quantising. The buffer is reallocated when a call needs more rows or a different K, dtype or device, and a captured graph keeps the old address, so run the largest M through it before capturing. ValueError if x is not 2-D or m_pad < M.

self_test()

op.self_test(m=None), collective: writes rank + 1 over the input region, runs scatter, reduce and gather without a GEMM, and checks every output element is P * (P + 1) / 2 within 1%; otherwise it raises RuntimeError naming BUILD_CCO_SDMA and MORI_ENABLE_SDMA. The logical m defaults to m_max, which compiles what the first real call needs. Call it once on every rank after construction and before any capture; it synchronises and overwrites the window.

close() and the context manager

op.close() deregisters the window and frees its memory. It is idempotent, and a closed op raises RuntimeError. Dropping the object is not enough, because the communicator holds the handles; the dev-comm and its SDMA queues stay with the communicator. Synchronise first if work may be in flight. with GemmAllReduceOp(...) as op: calls close() on exit.

Attributes

window_bytes (equal to window_bytes_for), m_max (padded) and the constructor’s settings are plain attributes, for reading. dev_comm.sdma_num_queue is the number of SDMA queues per peer the communicator actually connected.

7. Shapes and limits

supports(m, n, k, world_size, *, block_m=None, block_n=256, gather_dtype="bf16", gather_transport="lsa", quant="blockscale", scatter_dtype="bf16", schedule="default") takes a logical M and the constructor’s settings and returns False where these rules fail, without constructing anything; it does not apply the block_n == 256 rule or the bf16-scatter rule. It says whether a shape is expressible, not whether fusing it is faster.

Constraint

Value

Error if violated

quant

"blockscale", "mxfp8", "mxfp8_raw", "bf16"

ValueError: quant=... is not one of [...]

block_m

256 for the mxfp8 quants; 128 or 256 for "bf16"; a positive multiple of 128 for "blockscale"

ValueError: quant='mxfp8' requires block_m=256 ..., quant='bf16' requires block_m in (128, 256), or unsupported tile: block_m=... must be a positive multiple of 128

block_n

256

ValueError: block_n=... must be 256: the permlane epilogue's lane transpose pairs exactly two N-tiles

Wire and transport strings

dtypes "bf16", "fp8"; transports "lsa", "sdma"

ValueError: scatter_dtype=... is not one of ... (likewise the others)

Scatter wire with quant="bf16"

"bf16"

ValueError: quant='bf16' supports only the bf16 scatter wire

schedule

"default", "wo_b"

ValueError: unknown schedule=...; use 'default' or 'wo_b'

m_max, sdma_queues, max_shapes

at least 1

ValueError: m_max and sdma_queues must both be >= 1, max_shapes must be >= 1

K, fp8 quants

a multiple of 128, at least 256

ValueError: unsupported shape: K=... must be a multiple of 128 ... or ... is below the minimum 256

K, "bf16"

a multiple of 64, at least 128

ValueError: unsupported shape: K=... must be a multiple of 64 and >= 128

N

a multiple of 256

ValueError: unsupported shape: N=... must be a multiple of block_n=256

N, gather_dtype="fp8"

a multiple of 1024

ValueError: the fp8 wire gives one wave one row ... must be a multiple of 1024

Ranks P

2, 4 or 8 (in [2, 8] and dividing 512)

ValueError: world_size must be in [2, 8], threads (512) must be divisible by world_size ...

Call M

op.padded_m(logical), at most op.m_max

ValueError: M=... does not match planned padded M=...; use padded_m()/pad_rows() and pass the original logical_m, M=... exceeds the window's m_max=...

Distinct M plans

at most max_shapes

ValueError: this op already serves ... distinct M values ...; raise max_shapes, or pad to fewer distinct M

A and B

exact shape; dtype per quant (float8_e4m3fnuz refused); on the GPU

ValueError: a_fp8 must be (M, K), got ..., a_fp8 must be one of ['torch.float8_e4m3fn'], got ..., ... must be on a GPU

Scales

dtype, element count and layout per quant

ValueError: a_scale must be torch.float32, got ..., ... must have ... elements, a_scale is (M, K/128) row-major, which is ambiguous ..., quant='bf16' takes no scales; pass None

Lifetime

not closed

RuntimeError: this GemmAllReduceOp has been closed

Window

fits the free VMM reservation

RuntimeError: ccoMemAlloc failed: ... (logged: slot exhausted ... Increase perRankVmmSize)

8. Choosing settings

Setting

What the default picks

Change it when

quant

"blockscale"

the checkpoint is mxfp8 or BF16. "mxfp8_raw" takes the quantiser’s bytes with no shuffle; GEMM-only it measured faster cold and at large M, a few percent slower hot at small M or short K, and the two are not compared at K of 4096 and more

block_m

the quant’s tile: 128 for blockscale, 256 otherwise

rarely: blockscale takes other granule-legal tiles, and bf16 also takes 128, which doubles the grid at small M; the mxfp8 forms need 256

Chunks

the largest divisor of the bands per destination, up to 8

not a parameter; schedule="wo_b" changes it at measured M

schedule

"default": pad to P * block_m

V4-Pro TP8 or V4.1-Flash TP4 wo_b on MI355X (below)

gather_dtype

"bf16", exact

the accuracy budget allows relL2 about 2.5e-2 for a gather half the size

gather_transport

"lsa"

keep: the pull measured faster than the push on both targets

scatter_dtype

"bf16"

V4.1-Flash TP4, once accuracy is checked: faster there, slower on V4-Pro TP8

sdma_queues, max_shapes

1; every padded M up to m_max

keep

The fp8 gather halves the bytes of the leg that runs at the xGMI ceiling. At V4-Pro TP8, M=16384, the fused layer measured 957.3 µs with the LSA pull, 1018.9 µs with the SDMA push and 1150.7 µs on the bf16 wire; relL2 against an fp32 reference went from 2.35e-3 to 2.49e-2. At the model output it was not detectable: V4-Pro’s perplexity on 10941 tokens was 3.2671, inside the 3.2547 to 3.2680 of three bf16 runs. No reasoning, code or maths benchmark has been run.

schedule="wo_b" applies only on gfx950, to two targets. At P=8, N=7168, K=2048, "blockscale", block_m=128, both wires bf16: logical M=8200 keeps 9216 rows and pushes two bands at a time, and M=11264 and 13312 push one band at a time. At P=4, N=5120, K=2048, "mxfp8", block_m=256: M=4200 and 8200 pad to a multiple of 64 (4224, 8256) and push one band at a time, with both wires bf16, or with scatter_dtype="fp8" and a bf16 gather or an fp8 gather over LSA; with both wires bf16, M=11264 also pushes two bands at a time. Every other M, shape or setting keeps the default plan: it is a finite measured profile, not an interpolation rule.

Whether to fuse at your M. At V4.1-Flash TP4 against the model’s unfused path (section 9), the fp8 wire is ahead from M=2048 in every row, and the bf16 wire is behind at 1024, 2048, 4200 and 4700 and ahead from 7200. SGLang’s integration fuses that layer from M=8192 on the bf16 wire and from M=2048 on the fp8 wire, and V4-Pro’s from M=4096. Below that, run the GEMM (or Mxfp8GemmOp) and your usual all-reduce.

Environment variable

Default

Effect

MORI_ENABLE_SDMA

unset

Must be 1. Without it the communicator creates no SDMA queues and every put moves nothing.

MORI_GEMM_AR_PULL_BLOCKS

64

Grid of the LSA pull, so only with an fp8 gather over "lsa"; read when an M is compiled. At V4-Pro TP8, M=16384, 48 to 80 blocks measured 959 to 963 µs, against 1261 at 16 and 1184 at 512.

MORI_SOCKET_IFNAME

first non-loopback NIC

Interface the communicator’s bootstrap uses; the tests set lo.

9. Performance

All from the README, on MI355X (gfx950), in µs, with the conditions given for each table.

wo_b at M=16384, maximum over ranks. V4-Pro is on 8x MI355X, median of 11. V4.1-Flash is on 4x MI355X, from the README’s fp8-wire comparison; its fused bf16-wire cell comes from a different run and is left out here:

DeepSeek-V4-Pro (TP8, blockscale)

DeepSeek-V4.1-Flash (TP4, mxfp8)

split SDMA, bf16 wire

1472.9

1592.5

fused, bf16 wire

1148.8

—

fused, fp8 wire

979.3

1192.1 (over LSA)

GEMM + NCCL, as the model ships

1419.5

—

At smaller M on V4-Pro the order holds: 398.1 split, 351.0 fused and 329.5 fused with the fp8 gather at M=4096; 722.0, 621.1 and 539.3 at M=8192.

V4.1-Flash TP4 against what the model runs today, mxfp8_native_blockscaled_linear and then an all-reduce (the route is dot_scaled below M=8192 and hipblaslt_bf16 above). One M per process, two runs; fill is M over the padded M; negative is fused faster:

M

padded M

fill

today

fused, bf16 wire

fused, fp8 wire

1024

1024

1.000

164.6

+16.4%

+10.3%

2048

2048

1.000

283.3

+1.8%

-8.8%

4096

4096

1.000

482.8

-0.9%

-15.4%

4200

5120

0.820

506.1

+17.0%

-2.3%

4700

5120

0.918

548.3

+8.4%

-9.3%

5120

5120

1.000

601.5

-5.8%

-20.7%

7200

8192

0.879

1051.2

-17.6%

-32.7%

8192

8192

1.000

1042.5

-19.4%

-34.1%

8200

9216

0.890

1185.1

-20.8%

-35.7%

9200

9216

0.998

1157.5

-19.2%

-34.2%

13000

13312

0.977

1547.8

-8.5%

-25.3%

16384

16384

1.000

1851.3

-16.2%

-32.9%

The opt-in settings: prequantised operands, 8 calls per graph, 21 replays, 3 alternating rounds (5 for the short tails). The 2026-09-29 runs used Torch 2.11, ROCm 7.2.4 and FlyDSL 0.2.4; the 2026-09-30 timings are from prototype modules since integrated into the op:

Setting

Target

Date

Result

schedule="wo_b"

V4-Pro TP8

2026-09-30

M=11264 1074.35 → 819.42 (-23.7%); M=13312 1274.86 → 957.80 (-24.9%); M=8200 728.82 → 702.48

schedule="wo_b"

V4.1-Flash TP4

2026-09-29

reduced padding: M=4200 533.56 → 460.27 (-13.7%); M=8200 894.46 → 830.37 (-7.2%)

scatter_dtype="fp8"

V4.1-Flash TP4, M=16384

2026-09-29

-21.1% with a bf16 gather, -26.6% with the fp8 gather over LSA; relL2 2.62% and 3.48%

scatter_dtype="fp8"

V4-Pro TP8, M=16384

2026-09-29

3.0% and 2.5% slower

  • The two models do not reach the same conclusion. On V4-Pro the overlap dominates. On V4.1-Flash the GEMM is 178 µs against 1414 µs of communication, so most of the gain comes from the fp8 wire (-15.6% on its own) and the mxfp8 GEMM, not from fusing.

  • The fused cost follows the padded M while the baseline follows the true M, so one padded M can win at full fill and lose at low fill: read the whole curve. On Kimi-K3’s MLP down_proj (blockscale, bf16 wires, TP8 and TP4, against the op’s own split path) fusing loses 2.0-4.0% at M=1024 and gains 9.7-21.6% at M=16384.

  • In an SGLang server, V4.1-Flash prefill throughput at batch 4-16 rises +5.6% to +6.0% with the fused wo_b on the fp8 wire and +7.4% to +7.7% with the standalone mxfp8 GEMM added. V4-Pro’s GPU-busy time over a profiled prefill falls 5.0%. Layer gains exceed these because wo_b is about 12% of the profile.

10. Complete example

Run with MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=8 example.py. quantize_per_1x128 stands for your quantiser; MORI ships none. It must return a_fp8 as [m_pad, K] float8_e4m3fn and a_scale as fp32 1x128 block scales: [m_pad, K/128] column-major, [K/128, m_pad], or flat (section 5).

import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ar import GemmAllReduceOp, preshuffle_b

# DeepSeek-V4-Pro wo_b at TP8; add gather_dtype="fp8" for the shorter, lossy gather.
SETTINGS = dict(n=7168, k=2048, m_max=16384, quant="blockscale")
N, K = SETTINGS["n"], SETTINGS["k"]

torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo")  # only carries the unique id
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)

vmm = 2 * GemmAllReduceOp.window_bytes_for(world, **SETTINGS) + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
    with GemmAllReduceOp(comm, **SETTINGS) as op:  # the same settings as the sizing
        op.self_test()  # every rank, once, before any capture

        # This rank's weight shard; stand-ins for the checkpoint's tensors.
        w_fp8 = (torch.randn(N, K, device="cuda") / 8).to(torch.float8_e4m3fn)
        b_scale = torch.full((N // 128, K // 128), 1e-2, device="cuda")
        b = preshuffle_b(w_fp8)

        def wo_b(x):  # x: [M, K] bf16, M <= m_max
            m = x.shape[0]
            a_fp8, a_scale = quantize_per_1x128(op.pad_rows(x, op.padded_m(m)))
            return op(a_fp8, b, a_scale, b_scale)[:m]

        # Warm every M you will run (the first call compiles it), largest first.
        for m in (16384, 5000):
            y = wo_b(torch.randn(m, K, device="cuda", dtype=torch.bfloat16)).clone()

        # CUDA graph: a static input, warm-up on a side stream, then capture.
        x_static = torch.randn(16384, K, device="cuda", dtype=torch.bfloat16)
        side = torch.cuda.Stream()
        side.wait_stream(torch.cuda.current_stream())
        with torch.cuda.stream(side):
            for _ in range(3):
                wo_b(x_static)
        torch.cuda.current_stream().wait_stream(side)
        graph = torch.cuda.CUDAGraph()
        with torch.cuda.graph(graph):
            y_static = wo_b(x_static)  # a view of the window
        x_static.copy_(torch.randn_like(x_static))  # new data, same buffer
        graph.replay()  # on every rank, in the same order
        result = y_static.clone()

        torch.cuda.synchronize()  # nothing in flight when close() frees the window
dist.destroy_process_group()

11. Testing and benchmarking

# Host-only, single-GPU, 2-rank (public op), 8-rank (kernels) and TP8/TP4
# (wo_b schedule) tests; cases needing more GPUs than the node has are skipped
BUILD_CCO_SDMA=ON MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
  pytest tests/python/cco/test_gemm_ar.py tests/python/cco/test_flydsl_ar.py \
         tests/python/cco/test_gemm_ar_op.py tests/python/cco/test_gemm_ar_schedule.py

# DeepSeek-V4-Pro, TP8
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
  torchrun --standalone --nproc_per_node=8 benchmark/cco/flydsl/gemm_ar/bench_gemm_ar.py \
  --mode fused-sdma --quant blockscale -m 16384 -n 7168 -k 2048

# DeepSeek-V4.1-Flash, TP4, with the wire that wins there
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
  torchrun --standalone --nproc_per_node=4 benchmark/cco/flydsl/gemm_ar/bench_gemm_ar.py \
  --mode fused-sdma --quant mxfp8 -m 16384 -n 5120 -k 2048 \
  --gather-dtype fp8 --gather-transport lsa --no-fuse-quantize

# Matrices, one point per process; the exit status counts failed points
python benchmark/cco/flydsl/gemm_ar/sweep.py --list
python benchmark/cco/flydsl/gemm_ar/sweep.py fused      # writes fused.jsonl
python benchmark/cco/flydsl/gemm_ar/report.py fused.jsonl

tests/python/cco/test_gemm_ar_bench.py runs the benchmark as a test, and gemm_ar_schedule_worker.py beside it runs the wo_b regression directly. --mode takes gemm-only, split-rccl, split-sdma, fused-sdma, split-lsa or fused-lsa; each run prints a RESULT_JSON line with max_rank_time_us, rel_l2 and validated. Pin --quant (the default ptpc is not an op setting), and note --sdma-queues 8 and --gather-transport sdma against the op’s 1 and lsa. Allow about 10 s between back-to-back runs, and read the README’s Measurement traps before trusting a small-M number of your own.

12. Troubleshooting

Symptom

What it means

RuntimeError: gemm_ar self-test failed: the all-reduce returned ... where ... was expected

The SDMA puts moved nothing: MORI was built with BUILD_CCO_SDMA=OFF, or MORI_ENABLE_SDMA=1 is not set (the communicator also logs no SDMA queues will be created), and op.dev_comm.sdma_num_queue is 0. Without self_test() this is silent: the model still answers fluently and the fused path measures faster than it is.

ModuleNotFoundError: No module named 'flydsl' on first use of the op

Install the extra: pip install 'amd_mori[flydsl]'.

RuntimeError: ccoMemAlloc failed while constructing

The window does not fit the communicator’s VMM reservation. Size per_rank_vmm for every op alive at once, or close ops you no longer need.

hsaKmtCreateQueueExt failures (anvil.cpp) at start-up

A previous run’s ranks have not released their SDMA queues: wait about 10 s between runs (the tests wait 20 s) and check that no leftover process holds them. Keep sdma_queues=1 in a process that already uses SDMA.

ValueError: M=... does not match planned padded M=...

M is not padded: use op.padded_m() and op.pad_rows(), and with schedule="wo_b" pass logical_m=.

ValueError: a_scale is (M, K/128) row-major, which is ambiguous

Say which you have: .reshape(-1) for a K/128-major buffer, .t() for a logical row-major tensor. Guessing wrong pairs every scale with the wrong K block (relL2 0.26).

ValueError: ... must be one of ['torch.float8_e4m3fn'], got torch.float8_e4m3fnuz

FNUZ has a different exponent bias; read as OCP e4m3 the product would be 4x too large. Quantise to float8_e4m3fn.

ValueError: this op already serves ... distinct M values

An explicit max_shapes is too small: leave it at None, or pad to fewer distinct M.

A call hangs

The phases carry device-side barriers: every rank must call with the same M, in the same order. Two unexplained hangs were seen at 8 ranks while a sweep was being set up; use timeout when sweeping.

hipErrorIllegalAddress under sustained load

Seen once, in one SGLang server out of nine, with the fused wo_b and the standalone GEMM both enabled; it has not reproduced and the cause is not established.

Standalone GEMM and GEMV

Mxfp8GemmOp is the same mxfp8 GEMM with nothing fused; Mxfp8GemvOp is a separate skinny kernel for decode. Neither takes a communicator, a window or a rank or needs SDMA (both need FlyDSL), and both return an ordinary tensor. They are for a ColumnParallelLinear, which has no collective to fuse with, and for a RowParallelLinear below the fusion’s useful M.

Mxfp8GemmOp(*, n, k, block_n=256, quant="mxfp8")

Mxfp8GemvOp(*, n, k)

Call

op(a_fp8, b_preshuffled, a_scale, b_scale, out=None), [M, N] bf16

op(x_fp8, w_preshuffled, x_scale, w_scale, out=None), [M, N] bf16

M

"mxfp8": a multiple of 64 (op.padded_m(), op.pad_rows() before quantising); "mxfp8_raw": any

1 to 32; above, ValueError

Weight

preshuffle_b(w), shared with the GEMV, so shuffle once

the same tensor

Scales

as the fused op’s "mxfp8" or "mxfp8_raw" (section 5)

uint8 [M, K/32] and [N/32, K/32], 2-D, as the checkpoint stores them

Shapes

K a multiple of 128, at least 256; N a multiple of 256; block_n 256 only; quant "mxfp8" or "mxfp8_raw"

K a multiple of 128; N a multiple of 32

Predicate

supports_gemm(n, k, *, block_n=256)

supports_gemv(n, k)

Kernels

two: the 256-wide N tile, and a 128-wide one while the wide grid ceil(M/256) * N/256 is under 140 workgroups (op.wants_narrow_n(m))

one per M bucket (1, 2, 4, 8, 16, 32); tuned for (8192, 1280) and (5120, 2048), a heuristic elsewhere

out=, if given, must be [M, N] bf16. On MI355X at V4.1-Flash wo_b (N=5120, K=2048), M=16384, bf16 in and out with quantisation, the GEMM measured 196.0 µs against 281.3 µs for bf16 (fake_quant + hipBLASLt), -30.3%, and 274.9 µs for SGLang’s mxfp8 route (tl.dot_scaled); that was hot, one call per graph, and bench_gemm.py --scope linear puts the comparison at -27%. Cold, across V4.1-Flash’s fp8 linear shapes against SGLang’s native mxfp8 linear, what decides it is the wide-tile grid ceil(M/256) * N/256: the median is +246.7% at 1-32 workgroups, -0.7% at 33-64, -10.2% at 65-128, -26.6% at 129-256 and -26.4% above. On its two tuned shapes the GEMV beats SGLang’s mxfp8_gemv on the same fp8 bytes at M=1, 8 and 32; six untuned shapes land within ±2%, wo_a is 12-16% slower at M=8 and above, and wq_b at TP8 is slower at M=32.

Further reading

  • The gemm_ar README, next to the code, has the full design, every measurement table and the negative results.

  • MORI-FUSED overview: what the six operators share, and the headline numbers.