GEMM + all-gather

GemmAgOp (mori.ops.gemm_ag) computes C = A @ B.T on each rank’s own rows and gathers the results, so every rank gets every rank’s rows, [P*M, N]. It serves a projection on a context- or sequence-parallel token shard whose output every rank consumes, and was built for DeepSeek-V4’s compressor wkv_gate. The GEMM writes C into this rank’s receive slot. On the fused path, which the default picks at large M, its epilogue pushes each finished chunk of rows to every peer over the SDMA copy engines while the GEMM computes the rest.

Table of Contents

Quick Reference

import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ag import GemmAgOp

rank, world = dist.get_rank(), dist.get_world_size()   # after torch.cuda.set_device()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)

N, K, M_MAX = 2048, 7168, 16384          # DeepSeek-V4-Pro wkv_gate; M_MAX is rows per rank
vmm = 2 * GemmAgOp.window_bytes_for(world, n=N, m_max=M_MAX, out_dtype="fp32") + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
    with GemmAgOp(comm, n=N, k=K, m_max=M_MAX, in_dtype="bf16", out_dtype="fp32") as op:
        op.self_test()                    # collective, once
        m = op.padded_m(x.shape[0])       # x: this rank's [rows, K] bf16; the same m on every rank
        c = op(op.pad_rows(x, m), w)      # w: [N, K] bf16, the same on every rank
        # c: [world*m, N] fp32; source s owns rows [s*m, (s+1)*m); padding rows are zero

What

Import

Operator

from mori.ops.gemm_ag import GemmAgOp

Shape check

from mori.ops.gemm_ag import supports

Why a shape is rejected

from mori.ops.gemm_ag.op import shape_constraint

fp8 weight layout

from mori.ops.gemm_ag import preshuffle_b (the same function as mori.ops.gemm_ar.preshuffle_b)

mxfp8 A-scale layout

from mori.ops.gemm_ar import preshuffle_a_scale

Communicator

from mori.cco import Communicator, UniqueId

Window layout and kernel builders

from mori.ops.gemm_ag import ag_config, compile_bf16_gemm_ag, ... (section 6)


1. What it computes

Every rank r passes its own A_r [M, K] and the same B [N, K], and gets back C = concat(A_0 @ B.T, ..., A_{P-1} @ B.T), [P*M, N], with rank r’s rows at [r*M, (r+1)*M), its own included. P is the number of ranks and M the padded rows per rank, the same on every rank. B must be the same on every rank; the operator does not check it.

in_dtype

a

b

Scales

Result (and wire)

"bf16"

[M, K] bf16

[N, K] bf16, plain row-major

none

[P*M, N] bf16 or fp32 (out_dtype)

"fp8" (default)

[M, K] float8_e4m3fn

[N, K] float8_e4m3fn through preshuffle_b

by quant: ptpc, blockscale or mxfp8

[P*M, N] bf16

The layer is a projection on a context-parallel (CP) or sequence-parallel token shard whose output every rank needs: in DeepSeek-V4’s wkv_gate each CP rank scores its own tokens, then every rank needs every token’s score. The README’s model shapes are DeepSeek-V4-Pro’s ratio-4 compressor.wkv_gate (K=7168, N=2048, bf16 in, fp32 out) and DeepSeek-V4.1-Flash’s ratio-2 KV and gate projections (K=5120, N=512 each, or 1024 combined); M is the runtime token count per rank. SGLang’s Flash low-ratio CP path currently gathers the hidden states before the projection, so for Flash this operator is a reordered candidate.

2. How it works

Each call launches two kernels on the current stream. The GEMM writes C straight into this rank’s slot of the receive buffer; every peer receives that same contiguous slab, so nothing is staged.

  • split: a one-block kernel then pushes the slab to every peer, one SDMA put each, waits for the copies and runs a cross-rank barrier.

  • fused: the rows are cut into chunks of whole row tiles, one chunk per workgroup round of the GEMM grid, up to max_chunks. When a chunk’s last tile is stored, the epilogue posts the chunk to every peer. Two chunks for one peer use different SDMA queues, so a fused call needs a queue per chunk per peer. The second kernel only waits for the queues and runs the barrier.

  • auto (the default) picks one per M (section 8).

Fusion hides a chunk’s transfer behind the GEMM rounds still to come. At these shapes the transfer is several times the GEMM (Pro M=2048: 69 µs of GEMM against about 300 µs of all-gather), so fusion pays only at large M.

Receive-reuse credit. The result is the receive buffer, and the next call refills it. Each call first tells every peer it has started, and each push waits until its destination has started too, so a rank a call ahead never overwrites rows a slower rank is still reading. That holds provided whatever reads the result is ordered before this rank’s next call.

The window holds a few KiB of flags, chunk counters and credits, then the receive buffer, P * m_max * N elements of out_dtype; it is allocated once and cannot grow. A rank sends (P-1) * M * N * elem_bytes per call, 112 MiB at Pro M=2048 with fp32 output and P=8. The wire carries the output dtype, so fp32 doubles the bytes of bf16.

3. Requirements

The contract every fused operator shares (build and environment, window sizing, padding, result aliasing, the collective contract, cleanup, fp8 layouts) is in What every operator has in common. For this operator:

  • GPU: MI355X (gfx950). The GEMMs are FlyDSL kernels written for gfx950 and every measurement is on MI355X; the operator does not check the architecture. It reads the current device’s CU count, so call torch.cuda.set_device() before constructing it.

  • FlyDSL: pip install 'amd_mori[flydsl]'. import mori.ops.gemm_ag, preshuffle_b and the layout helpers work without it; GemmAgOp, supports and the kernel builders import it on first use.

  • SDMA: a MORI built with BUILD_CCO_SDMA=ON (the default), and MORI_ENABLE_SDMA=1 in the environment when the communicator is created, which is when MORI reads it.

  • SDMA queues: mode="auto" and "fused" ask for max_chunks per peer (default 4), "split" for 1. The first device communicator on a communicator fixes its queue count, later ones reuse it, and MORI creates at most 8 per pair of GPUs. The constructor raises RuntimeError if there are too few.

  • Ranks: 2 to 8, all in one node (one LSA domain).

  • VMM: the window comes out of the communicator’s per_rank_vmm; size it with window_bytes_for (section 4) plus every other window on the communicator. The examples reserve 2 * window_bytes_for(...) + (256 << 20), as the overview does.

4. Constructor

GemmAgOp(comm, *, n, k, m_max, in_dtype="fp8", quant="ptpc", out_dtype="bf16",
         mode="auto", max_chunks=4, chunks=None, tile=None, fence="release",
         peer_uncached=True, war_credit=True)

Collective: every rank constructs with the same arguments. It validates them, allocates and zeroes the window, creates a device communicator with its SDMA queues, and ends with a barrier.

Name

Type

Default

Description

comm

mori.cco.Communicator

required

Its ranks are the all-gather’s ranks. The only positional argument.

n

int

required

Output columns N, the weight’s rows. A positive multiple of 256.

k

int

required

Inner dimension K. fp8: a multiple of 128, at least 256. bf16: a multiple of 64, at least 128.

m_max

int

required

The most rows per rank any call will pass, at least 1. Rounded up to the M granule (op.m_max holds the result); sizes the window.

in_dtype

str

"fp8"

"fp8" (float8_e4m3fn operands, gemm_ar’s 8-wave fp8 GEMM) or "bf16".

quant

str

"ptpc"

fp8 scaling: "ptpc" (per token, per channel), "blockscale" (A 1x128, B 128x128) or "mxfp8" (32-wide ue8m0). With in_dtype="bf16" leave it at "ptpc".

out_dtype

str

"bf16"

Result and wire dtype, "bf16" or "fp32"; "fp32" needs in_dtype="bf16".

mode

str

"auto"

"fused", "split", or "auto": fused where ceil(M/256) * ceil(N/256) is at least the device’s CU count.

max_chunks

int

4

The most chunks a fused call pushes in, and the SDMA queues per peer that "auto" and "fused" ask for. At least 1. Pass the same value to window_bytes_for.

chunks

int or None

None

Pins the fused chunk count instead of one per workgroup round; rounded down to a divisor of the row tiles, capped by max_chunks. At least 1.

tile

tuple[int, int] or None

None

Pins (BLOCK_M, BLOCK_N) for every M; M then pads to BLOCK_M. bf16: each 128 or 256, BLOCK_N=128 only with fp32 output. fp8: BLOCK_N=256, BLOCK_M 128 or 256 (256 under mxfp8).

fence

str

"release"

The fused epilogue’s fence before it pushes a chunk: "release" (one release-only system fence per workgroup), "leader" (a system fence from one wave per workgroup), "all" (from every lane) or "none" (no fence; gemm_ar measured the copy engine then reading some tiles stale on 8 ranks).

peer_uncached

bool

True

Fused calls store C uncached (sc0, sc1). With fence="release" this is the README’s sc_release, used by every fused number there.

war_credit

bool

True

The receive-reuse credit. False exists only for the race test; back-to-back calls can then return another call’s rows.

chunks, fence and peer_uncached affect fused calls only.

window_bytes_for()

GemmAgOp.window_bytes_for(world_size, *, n, m_max, out_dtype="bf16", max_chunks=4) -> int

Static: the most bytes the constructor will allocate for its window, for sizing the communicator’s VMM before the operator exists. world_size (2 to 8) is the number of ranks; pass n, m_max, out_dtype ("fp32" doubles the receive buffer) and max_chunks (it sizes the chunk counters) as the constructor gets them. It rounds m_max up to 256, the largest M granule, so the value holds for every in_dtype, quant, mode and tile. Raises ValueError for a world size outside 2-8, an m_max below 1, or an n that is not a positive multiple of 256.

5. Calling the operator

out = op(a, b, a_scale=None, b_scale=None)    # -> torch.Tensor [P*M, N]

One GEMM + all-gather, issued on the current stream; it returns without synchronising. M = a.shape[0] must be a positive multiple of the M granule (use padded_m), at most op.m_max, and the same on every rank. The first call at each M compiles that M’s kernels.

Argument

in_dtype="bf16"

in_dtype="fp8"

a

[M, K] torch.bfloat16

[M, K] torch.float8_e4m3fn (float8_e4m3fnuz is rejected)

b

[N, K] torch.bfloat16, plain row-major

[N, K] torch.float8_e4m3fn through preshuffle_b

a_scale, b_scale

None

required, by quant (below)

quant

a_scale

b_scale

"ptpc"

[M] fp32, one per row

[N] fp32

"blockscale"

fp32 per 1x128 block: [K/128, M], [M, K/128] column-major, or flat in [K/128, M] order

[N/128, K/128] fp32

"mxfp8"

the flat int32 buffer preshuffle_a_scale(a_exps) returns, a_exps being the [M, K/32] ue8m0 bytes

the [N/32, K/32] ue8m0 bytes w_exps K-block-major, one per int32: w_exps.t().contiguous().to(torch.int32).reshape(-1)

All operands must be GPU tensors; a and b are made contiguous (a copy if they are not). A row-major [M, K/128] blockscale a_scale is rejected as ambiguous: pass a_scale.reshape(-1) if its storage is already [K/128, M] (as aiter_per1x128_quant(transpose_scale=True) returns it), or a_scale.t() for a logical [M, K/128].

Return value: a contiguous [P*M, N] tensor in out_dtype, rank s’s rows at [s*M, (s+1)*M). Zero-padded input rows come back as zero rows in every source’s block; slice them off per source. It is a view of the operator’s window, valid until this rank’s next call, self_test() or close(). Order whatever reads it before that call (same stream, or an event the next call’s stream waits on), or clone it.

6. Other methods

padded_m()

op.padded_m(m) -> int rounds a row count up to op.m_granule: 128, 256 under quant="mxfp8", or BLOCK_M when tile is pinned. Pad from a count that is the same on every rank. ValueError if m < 1.

pad_rows()

op.pad_rows(x, m_pad) -> torch.Tensor zero-extends a 2-D [m, K] tensor to [m_pad, K] in a buffer the operator reuses while m_pad fits and K, dtype and device stay the same, so the next pad_rows overwrites it and a CUDA graph that reads it sees each refill. Pad before quantising, so fp8 scales cover the padded rows. ValueError if x is not 2-D or m_pad is too small.

fuses_at()

op.fuses_at(m) -> bool: whether a call at this M runs the fused path. Under mode="auto" that is ceil(M/256) * ceil(N/256) >= op.cus; under "fused" or "split" it is always True or False. No GPU work.

tile_for()

op.tile_for(m) -> tuple[int, int]: the (BLOCK_M, BLOCK_N) a call at this padded M uses: the pinned tile, else for bf16 the per-shape rule in section 8, else 128x256 (256x256 under mxfp8).

chunks_for()

op.chunks_for(m) -> int: the chunks a call at this padded M pushes in. 1 on the split path; otherwise the pinned chunks or one per workgroup round of the tile, capped at max_chunks and rounded down to a divisor of the row tiles.

self_test()

op.self_test() fills this rank’s slot with rank + 1, runs the split push at m_max and checks every peer’s slot, raising RuntimeError at the first wrong one; a put that silently moves nothing fails here. A collective: call it on every rank once after construction, before capturing a graph. It synchronises the device and overwrites the receive buffer.

close() and the context manager

op.close(), or leaving a with GemmAgOp(...) as op: block, releases the window and drops the compiled plans and the pad_rows buffer. Idempotent. Synchronise first if a call may still be in flight. Afterwards a call or self_test() raises RuntimeError: this GemmAgOp has been closed.

Attributes

The constructor also sets, for reading: m_max (rounded up to the granule), m_granule, window_bytes (this window’s size), sdma_queues (max_chunks, or 1 under mode="split"), cus (the CU count the auto, tile and chunk rules use, read at construction), rank and world_size.

supports()

supports(n, k, world_size, **kwargs) -> bool is True when 2 <= world_size <= 8 and shape_constraint(n, k, **kwargs) finds nothing; kwargs are in_dtype, quant and out_dtype, with the constructor’s defaults. It says whether a shape is expressible, not whether it is faster, and looks at neither M, the GPU nor SDMA.

shape_constraint()

shape_constraint(n, k, *, in_dtype="fp8", quant="ptpc", out_dtype="bf16") -> str | None returns why the constructor would reject this N, K and precision (it raises ValueError("unsupported configuration: " + reason)), or None. Import it from mori.ops.gemm_ag.op, which also exports the allowed values as IN_DTYPES, QUANTS, OUT_DTYPES, MODES and FENCES.

preshuffle_b()

preshuffle_b(w) -> torch.Tensor reorders an fp8 [N, K] weight into the order the fp8 GEMM reads; shape and dtype are unchanged. Call it once per weight. It needs a 1-byte dtype, N a multiple of 16 and K a multiple of 64 (ValueError otherwise). bf16 weights are not shuffled.

Kernel-level exports

mori.ops.gemm_ag also exports what GemmAgOp is built from. AgConfig, ag_config, counter_chunks and MAX_WORLD (8) are the window layout and launch arithmetic, importable without FlyDSL. compile_bf16_gemm_ag, compile_fused_gemm_ag, compile_gemm_local, build_sdma_phases and build_sdma_chunk_post build the kernels. The benchmarks call them directly; the operator’s users do not need them. build_sdma_chunk_post serves a two-stream pipeline experiment and carries no receive-reuse credit.

7. Shapes and limits

Constraint

Value

Error if violated

in_dtype, quant, out_dtype

as in section 4

ValueError: unsupported configuration: in_dtype=... is not one of ... (likewise quant, out_dtype); ... quant does not apply to in_dtype='bf16'; leave it at its default; ... the fp8 GEMM's epilogue only emits bf16

N

a positive multiple of 256

ValueError: unsupported configuration: N=... must be a positive multiple of 256

K, fp8

a multiple of 128, at least 256

ValueError: unsupported configuration: K=... must be a multiple of 128 and at least 256

K, bf16

a multiple of 64, at least 128

ValueError: unsupported configuration: K=... must be a multiple of 64 and at least 128

mode, fence

the listed values

ValueError: mode=... is not one of ..., fence=... is not one of ...

m_max, max_chunks, chunks

at least 1 (chunks may be None)

ValueError: m_max and max_chunks must be >= 1; ..., chunks must be >= 1, got ...

tile, fp8

BLOCK_N=256, BLOCK_M 128 or 256 (256 under mxfp8)

ValueError: tile=...: the fp8 GEMM takes BLOCK_N=256 and BLOCK_M 128 or 256 (256 under mxfp8)

tile, bf16

each 128 or 256, BLOCK_N=128 only with fp32 output

ValueError from the window layout: block_m=... must be a positive multiple of 128 ..., block_n must be 128 or 256 ..., block_n=128 needs elem_bytes=4 (an fp32 C) .... A BLOCK_M of 384 or more passes construction and fails the first call with BLOCK_M must be 128 or 256 ...

Ranks

2 to 8

ValueError: world_size must be in [2, 8], got ... (also from window_bytes_for)

One node

every rank in one LSA domain

RuntimeError: GemmAgOp needs every rank in one LSA domain (one node); ...

SDMA queues

at least sdma_queues per peer

RuntimeError: GemmAgOp needs ... SDMA queue(s) per peer but the communicator has ...

Open (each call)

not closed

RuntimeError: this GemmAgOp has been closed

M (each call)

a positive multiple of op.m_granule, at most op.m_max

ValueError: M=... must be a positive multiple of ...; use padded_m() and pad_rows(); M=... exceeds the window's m_max=...

a, b (each call)

[M, K] and [N, K], the dtype of in_dtype, on a GPU

ValueError: a must be (M, K), got ...; ... must be one of [...] for in_dtype=..., got ...; the launch reinterprets it as raw bytes; ... must be on a GPU, got ...

Scales (each call)

none for bf16; both for fp8

ValueError: in_dtype='bf16' takes no scales; quant=... needs a_scale and b_scale

Scale dtype and size (each call)

as in section 5

ValueError: a_scale must be torch.float32, got ...; a_scale must have ... elements, got ...; b_scale must have ... elements, got ...; the layout messages for blockscale and mxfp8

8. Choosing settings

The defaults were tuned on MI355X for the bf16-in, fp32-out wkv_gate shapes; leave them unless a measurement of your own says otherwise.

  • mode="auto" fuses once even the largest tile’s grid covers the CUs, ceil(M/256) * ceil(N/256) >= CUs: a GEMM long enough for the transfer to hide behind. The rule fits every pair in the README’s results table at 256 CUs; a GPU with more CUs needs a larger M. Pin "split" when M stays small or SDMA queues are scarce (it asks for one per peer).

  • bf16 tile, per M and per path, among 256x256, 128x256, 256x128 and 128x128. Split: the largest tile whose grid still covers the CUs, else the largest grid. Fused: the largest tile other than 256x256 whose grid runs at least two workgroup rounds, else the largest grid. fp8 uses 128x256 (256x256 under mxfp8).

  • Chunks: one per workgroup round of the fused tile, capped at max_chunks. The cap defaults to 4 because 4 measured within 2% of 8 where the rule would pick 8, and a queue per chunk per peer is what runs out of SDMA engine slots first.

  • fence, peer_uncached: the defaults are the README’s sc_release. On 2026-10-29 the benchmark’s default fence (leader) measured within 0.2% of it at M >= 4096, and 1.3% to 2.5% slower at M=2048. Do not use "none".

  • war_credit: leave it on.

  • No max_shapes: one instance serves any number of distinct M up to m_max, in any order. Each new M compiles once; warm every M before capturing a CUDA graph.

What the defaults pick on MI355X (256 CUs), bf16 in, fp32 out, as fuses_at, tile_for and chunks_for report it:

Shape

M

Path

Tile

Chunks

Pro wkv_gate, N=2048

2048

split

128x128

1

4096

split

128x256

1

8192

fused

128x256

2

16384

fused

128x256

4

Flash combined, N=1024

2048

split

128x128

1

16384

fused

128x256

2

Environment variables. The operator reads none itself. MORI reads MORI_ENABLE_SDMA when the communicator is created; MORI_SOCKET_IFNAME names the network interface of the communicator’s bootstrap (the tests use lo).

9. Performance

From the gemm_ag README, on MI355X, in µs. Each table is its own session; compare within a table only.

Through the operator: Pro wkv_gate (N=2048, K=7168), bf16 in, fp32 out, graph-timed, maximum over ranks. sc_release is the default publication; leader is the leader fence.

P

M

auto

fused (sc_release)

fused (leader)

split

8

2048

351.3 (split)

362.8

361.0

351.3

8

4096

666.2 (split)

685.5

687.6

666.1

8

8192

1239.2 (fused)

1240.0

1240.9

1288.7

4

2048

349.1 (split)

357.0

358.5

349.4

4

4096

662.0 (split)

682.8

683.3

661.7

4

8192

1231.3 (fused)

1231.3

1232.7

1282.2

Against Torch + RCCL: 2026-10-29, eight MI355X, ROCm 7.2, PyTorch 2.11.0+rocm7.2, FlyDSL 0.4.0; the benchmark’s modes, bf16 in, fp32 out, MORI’s GEMM at a fixed 128x128 tile; 50 warm-ups and 101 graph samples, maximum over ranks, median of three interleaved rounds. Fused rows use sc_release at the best chunk count (cN).

Shape

M

Torch + RCCL

MORI GEMM + SDMA

best fused SDMA

best vs Torch + RCCL

Pro wkv_gate, N=2048 K=7168

2048

415.1

373.4

376.6 (c1)

-10.0%

4096

823.2

705.6

716.8 (c2)

-14.3%

8192

1510.5

1381.7

1314.8 (c4)

-13.0%

16384

3008.2

2844.2

2527.7 (c8)

-16.0%

Flash combined, N=1024 K=5120

2048

227.6

213.6

213.5 (c1)

-6.2%

16384

1465.9

1372.7

1309.2 (c4)

-10.7%

How to read them:

  • Fusion pays from M=8192 on Pro and at M=16384 on Flash N=1024, not below; mode="auto" encodes that and takes the faster path in every row of the first table. At its own tile (128x256, a chunk per round) the fused path measured 3.9%, 7.4% and 2.2% under split SDMA at Pro M=8192, Pro M=16384 and Flash M=16384, on the box of the 2026-10-29 table.

  • Against Torch + RCCL the steady part of the gain is the transport: SDMA’s all-gather alone is 12.5-13.9% faster than RCCL’s at every shape. The fixed 128x128 tile falls behind Torch’s GEMM at large M; the per-M tile closes that from M=8192.

  • The README’s only fp8 measurement, an earlier protocol at M=N=2048, K=7168, has one-chunk fusion behind split SDMA under ptpc, blockscale and mxfp8.

10. Complete example

bf16 in, fp32 out, the Pro wkv_gate shape, replayed from a CUDA graph:

# MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo torchrun --standalone --nproc_per_node=8 example.py
import os

import torch
import torch.distributed as dist

from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ag import GemmAgOp

N, K, M_MAX = 2048, 7168, 16384   # DeepSeek-V4-Pro wkv_gate; the most rows per rank ever passed
TOKENS = 3000                     # rows per rank this step, the same on every rank

torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo")
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)

vmm = 2 * GemmAgOp.window_bytes_for(world, n=N, m_max=M_MAX, out_dtype="fp32") + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
    with GemmAgOp(comm, n=N, k=K, m_max=M_MAX, in_dtype="bf16", out_dtype="fp32") as op:
        op.self_test()                                     # collective, once

        g = torch.Generator(device="cuda").manual_seed(0)  # the same weight on every rank
        w = (torch.randn(N, K, generator=g, device="cuda") / 8).to(torch.bfloat16)
        torch.manual_seed(1 + rank)                        # a different shard per rank
        x = (torch.randn(TOKENS, K, device="cuda") / 8).to(torch.bfloat16)

        m = op.padded_m(TOKENS)                            # 3072
        a = op.pad_rows(x, m)                              # the buffer the graph reads
        print(rank, op.fuses_at(m), op.tile_for(m), op.chunks_for(m))  # MI355X: False (128, 128) 1

        op(a, w)                                           # warm-up: compiles this M
        torch.cuda.synchronize()
        graph = torch.cuda.CUDAGraph()
        with torch.cuda.graph(graph):
            out = op(a, w)                                 # [world*m, N] fp32, in the window

        for _ in range(3):                                 # the same count on every rank
            x = (torch.randn(TOKENS, K, device="cuda") / 8).to(torch.bfloat16)
            op.pad_rows(x, m)                              # refills `a` in place
            graph.replay()
            result = out.view(world, m, N)[:, :TOKENS].reshape(world * TOKENS, N)  # a copy
        torch.cuda.synchronize()                           # nothing in flight at close()
dist.destroy_process_group()

fp8 in, bf16 out (the constructor’s default precision) with ptpc scales: as above, but size the VMM for bf16 output (window_bytes_for(world, n=N, m_max=M_MAX)) and replace the operator block with

from mori.ops.gemm_ag import preshuffle_b

with GemmAgOp(comm, n=N, k=K, m_max=M_MAX, quant="ptpc") as op:   # in_dtype="fp8" by default
    op.self_test()
    b = preshuffle_b(w_fp8)                               # [N, K] float8_e4m3fn, once per weight
    m = op.padded_m(TOKENS)
    a, a_scale = quantize_per_token(op.pad_rows(x, m))    # your quantiser
    out = op(a, b, a_scale, w_scale)                      # [world*m, N] bf16

MORI ships no quantiser. quantize_per_token must return a as [m, K] torch.float8_e4m3fn and a_scale as [m] fp32, one scale per row including the padded rows; w_scale is the weight’s [N] fp32 scale. For blockscale and mxfp8, produce the layouts in section 5.

11. Testing and benchmarking

Run against an installed MORI: the checkout’s python/ has no compiled mori.cco.cco, so keep it off PYTHONPATH.

export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest tests/python/cco/test_gemm_ag.py tests/python/cco/test_gemm_ag_op.py

test_gemm_ag.py covers the layout arithmetic, the tile and chunk rules, models of the LDS ordering and SDMA completion, and single-GPU GEMM numerics (skipped without gfx950). test_gemm_ag_op.py checks the configuration rules, then runs one 8-rank torchrun of gemm_ag_op_worker.py: every precision in both modes, every bf16 tile, chunk counts, back-to-back calls with a held reader with the credit on and off, alternating M, padding, operand checks, self_test and close. The multi-rank parts need 8 GPUs and FlyDSL.

The benchmark benchmark/cco/flydsl/gemm_ag/bench_gemm_ag.py calls the kernels, not GemmAgOp, and defaults to the leader fence with cached stores. With --block-m, --block-n and --chunks left at 0 it picks the tile and chunks by the operator’s rules at 256 CUs, with up to 8 chunks. Fused SDMA at the Pro shape, as the operator runs it:

python -m torch.distributed.run --standalone --nproc_per_node=8 \
  benchmark/cco/flydsl/gemm_ag/bench_gemm_ag.py \
  --mode fused-sdma --peer-uncached --fence release \
  --in-dtype bf16 --out-dtype fp32 -m 8192 --out-dim 2048 -k 7168 \
  --rounds 5 --warmup 50 --iters 101 --tolerance 1e-5

--mode also takes gemm-only, gemm-to-window, split-rccl and split-sdma (--fence release is refused outside fused-sdma). --in-dtype defaults to fp8, with --quant ptpc|blockscale|mxfp8 and a default --tolerance of 3e-3. --torch-gemm swaps in torch.mm for bf16-in, fp32-out gemm-only, split-rccl and split-sdma; --phase-split also times the GEMM alone. Each run prints RESULT_JSON and TIMING_JSON; the time is the median over rounds of the per-round maximum over ranks. Run one configuration at a time on an idle node, and let one process group’s SDMA queues tear down before the next. The benchmark README also covers the two-stream pipeline experiment and the GEMM-only comparison.

12. Troubleshooting

Error or symptom

What it means

RuntimeError: GemmAgOp needs 4 SDMA queue(s) per peer but the communicator has 0

No SDMA: MORI was built without BUILD_CCO_SDMA=ON, or MORI_ENABLE_SDMA=1 was not set when the communicator was created. MORI also logs no SDMA queues will be created (...) with the reason.

The same error with a non-zero count

An earlier device communicator on this communicator fixed a smaller queue count. Construct the operator that needs the most queues first, lower max_chunks, or use mode="split". More than 8 per peer is never available.

RuntimeError: GemmAgOp needs every rank in one LSA domain (one node)

The ranks span nodes; the operator is single-node.

RuntimeError: gemm_ag self-test failed: the slot from rank s holds ..., expected ...

The puts complete but move nothing. Check BUILD_CCO_SDMA and MORI_ENABLE_SDMA.

RuntimeError: ccoMemAlloc failed: ... in the constructor

Usually the window does not fit in what is left of per_rank_vmm. Size it with window_bytes_for for every window on the communicator.

ValueError: M=... must be a positive multiple of ...

Pad with padded_m() and pad_rows(). Under mxfp8 or a pinned 256-row tile the granule is 256.

ValueError: M=... exceeds the window's m_max=...

Construct with a larger m_max, and size the VMM for it.

ValueError: a must be one of ['torch.float8_e4m3fn'] ...

Wrong operand dtype, for example float8_e4m3fnuz, or bf16 on an fp8 instance.

ValueError: a_scale is (M, K/128) row-major, which is ambiguous ...

Pass a_scale.reshape(-1) for a buffer already in [K/128, M] order, or a_scale.t() for a logical [M, K/128].

Wrong numbers, no error

An fp8 b not passed through preshuffle_b, a b that differs between ranks, or scales in another layout. None of these shows in the shapes.

The result changes after the call

It was read after this rank’s next call (or self_test()), or on another stream without an event. Order the read before the next call, or clone it.

A hang

The ranks disagree on the collective contract: a different number of calls or graph replays, a different M, or only some ranks constructing or calling self_test().

hsaKmtCreateQueueExt or anvil.cpp errors at start-up

The SDMA queues of the previous multi-rank run are not released yet. Wait and retry; the tests wait for the previous run’s processes to leave the GPU, then retry once.

Further reading