All-to-all + GEMM

A2aGemmOp (mori.ops.a2a_gemm) fuses the all-to-all at the end of Ulysses attention into the fp8 GEMM of the o-projection. Every rank passes its head shard for every token, [P*M, K/P], and the same weight B [N, K], and gets back C [M, N] in bf16: its own M tokens over all of K. Block 0 of the GEMM posts every band of rows to every peer over the SDMA copy engines. The GEMM starts on this rank’s own K segment and reads each remote segment as soon as its band has landed. This is the reverse of GemmA2AOp.

Table of Contents

Quick Reference

The snippet assumes the setup in What every operator has in common: FlyDSL and SDMA, a communicator whose VMM reservation is sized with window_bytes_for, and one self_test() per operator.

from mori.cco import Communicator, UniqueId
from mori.ops.a2a_gemm import A2aGemmOp
from mori.ops.gemm_ar import preshuffle_b

# rank, world and uid (a mori.cco.UniqueId): see the complete example
vmm = 2 * A2aGemmOp.window_bytes_for(world, k=K, m_max=max_tokens_per_rank) + (256 << 20)
with Communicator.init(world, rank, uid, per_rank_vmm=vmm) as comm:
    with A2aGemmOp(comm, n=N, k=K, m_max=max_tokens_per_rank) as op:
        op.self_test()                             # collective; raises if no bytes move
        b = preshuffle_b(w_o)                      # [N, K] fp8, the same on every rank
        m = op.padded_m(tokens_per_rank)
        a = op.send_buffer(m)                      # [P*m, K/P] fp8, in the window
        quantize_attention_output_into(a)          # your quantiser
        c = op(a, b, a_scale, b_scale)             # [m, N] bf16; a_scale [m], b_scale [N]

What

Import

Operator

from mori.ops.a2a_gemm import A2aGemmOp

Weight layout

from mori.ops.gemm_ar import preshuffle_b

Communicator

from mori.cco import Communicator, UniqueId

Default bands and wait per M

from mori.ops.a2a_gemm import default_chunks, default_wait

Limits (MAX_WORLD 8, MAX_CHUNKS 16, BLOCK_K 128)

from mori.ops.a2a_gemm import MAX_WORLD, MAX_CHUNKS, BLOCK_K

Window layout and kernel builders, which the benchmark drives

from mori.ops.a2a_gemm import A2aGemmConfig, compile_a2a_gemm, build_phases

1. What it computes

Ulysses sequence parallelism over P ranks ends attention with every rank holding all P*M tokens for its own K/P columns of heads. The o-projection needs this rank’s M tokens over all of K, times the replicated weight B [N, K]. Rank d computes

C_d = concat_s(A_s[d*M:(d+1)*M]) @ B.T                  # [M, N]
C_d[i, j] = a_scale[i] * b_scale[j] * sum_k A[i, k] * B[j, k]

A_s is source rank s’s input. The concatenation is along K: source s owns columns [s*K/P, (s+1)*K/P), and A is the gathered [M, K] operand. Unfused, this is all_to_all_single followed by the GEMM.

Per rank

Shape

Dtype

Content

a_fp8 (input)

[P*M, K/P]

float8_e4m3fn

Every token, this rank’s head columns. Rows [d*M, (d+1)*M) are rank d’s tokens.

b_preshuffled

[N, K]

float8_e4m3fn

The o-projection weight through preshuffle_b, identical on every rank

a_scale

[M]

float32

One scale per token of this rank, over all of K

b_scale

[N]

float32

One scale per output channel

C (output)

[M, N]

bfloat16

This rank’s tokens, rows [rank*M, (rank+1)*M) of the sequence

fp8 ptpc is the only precision. The README measures the MiniMax-M3 o-projection at Ulysses SP4 and the Llama-3.1-70B o-projection at P=8.

2. How it works

The operator’s window keeps what this rank sends and what it receives side by side, in the same per-source layout:

[ control | send: [P][M][K/P] fp8 | recv: [P][M][K/P] fp8 ]

send[d], the rows bound for rank d, is one contiguous slab, and send_buffer(M) is the whole send region as [P*M, K/P]. recv[s] receives source s’s rows. The GEMM reads A one K segment at a time: send[rank] for this rank’s own columns, which never travel, and recv[s] for every other source. One call with mode="fused":

  1. Post. In block 0 of the GEMM, one lane per peer pushes send[peer] in chunks row bands, back to back, on the copy engine. Each band bumps a counter at the receiver once its bytes have landed.

  2. Compute. Tiles go band-major: the first tiles need only band 0 from each source, the next ones band 1, and so on. A tile starts on this rank’s own K segment, then takes the other sources in rank-rotated order.

  3. Wait. wait="segment": each wave waits for a segment’s band just before its first load of it, so the own segment overlaps the transfer. wait="tile": one thread waits for the tile’s band from every source before the tile starts. Block 0 takes the last tile, which needs the last band anyway.

  4. Close. A barrier-only kernel ends every call, so no rank refills a peer’s recv while that peer still reads it. It also advances the epoch and re-takes the counter snapshot the band waits count from, both kept in the window: the next call, at any M and band count, and a CUDA-graph replay start from there.

mode="split" runs a push kernel, a barrier, the GEMM with no waits and the closing barrier, over the same window. For the settings this means:

  • More bands let the first tiles start after less of the transfer, but each band has a fixed cost. On an idle GPU the README’s 70B exchange moves ~47 GB/s per link in one band and ~33 GB/s in eight.

  • With wait="segment", what a tile can do before remote bytes arrive is its own segment: a quarter of K at P=4 and an eighth at P=8, too little at P=8 to cover the wait for the first bands.

3. Requirements

The shared contract (build and environment, window sizing, padding, result aliasing, the collective contract, cleanup, fp8 operands) is in What every operator has in common. For this operator:

Requirement

Detail

GPU

gfx950 (MI355X). The GEMM is gemm_ar’s 8-wave fp8 FlyDSL kernel, written for gfx950 only, and every measurement is on MI355X. The operator does not check the architecture.

FlyDSL

pip install 'amd_mori[flydsl]' (Install MORI). import mori.ops.a2a_gemm works without it, for A2aGemmConfig and the constants. A2aGemmOp, default_chunks, default_wait and the kernel builders import FlyDSL.

SDMA

MORI built with BUILD_CCO_SDMA=ON (the default), and MORI_ENABLE_SDMA=1 set before the communicator is created. The operator asks for one SDMA queue per peer, and its constructor raises if the communicator has none. self_test() checks that bytes actually move.

Ranks

P from 2 to 8, all in one node (one LSA domain).

Communicator

A mori.cco.Communicator whose per_rank_vmm (4 GiB by default) holds the window: 2 * m_max * K bytes plus 4 KiB. Size it from window_bytes_for().

4. Constructor

A2aGemmOp(comm, *, n, k, m_max, mode="fused", chunks=None, wait=None, acquire="inv")

Name

Type

Default

Description

comm

mori.cco.Communicator

required

The communicator every rank shares: 2 to 8 ranks in one node, with SDMA queues. Its per_rank_vmm must hold the window.

n

int

required

N, the output columns and the rows of B [N, K]. A multiple of 256.

k

int

required

K, the full reduction dimension (every head). A multiple of P * 128, at least 256. Each rank owns K/P columns (op.k_shard).

m_max

int

required

The largest M (tokens per rank) the operator will see, at least 1. Rounded up to a multiple of 256 (op.m_max). It sizes the window, which cannot grow.

mode

str

"fused"

"fused": the GEMM posts the bands and each tile waits only for what it reads. "split": push, barrier, GEMM, barrier over the same window; the baseline and a fallback.

chunks

int or None

None

Bands per peer per call, 1 to 16. None picks a count per M with default_chunks. A number is an upper bound: at each M the operator uses the largest divisor of M/256 that does not exceed it. In "split" it sets the number of puts per peer.

wait

str or None

None

How a tile waits in "fused": "segment" (each wave, before its first load of a remote segment) or "tile" (once per tile, for every source, before the tile starts). None picks per M with default_wait. No effect in "split".

acquire

str

"inv"

What a tile does after a band has landed, in "fused". "inv": an acquire fence that invalidates the caches. "none": nothing, which is correct only while cco windows are uncached, the default (not with CCO_UNCACHED_WINDOW=0). The README’s tables use "none".

The constructor is collective: call it on every rank with the same arguments. It checks its arguments before allocating, then allocates and zeroes the window, creates a device communicator with one SDMA queue per peer, checks the queues and the LSA domain, and snapshots the copy engines’ signal counters between two host barriers. One instance serves any M up to m_max, and every layer with this N and K: the weight is a call argument.

window_bytes_for()

A2aGemmOp.window_bytes_for(world_size, *, k, m_max) -> int is a static method.

Name

Type

Description

world_size

int

P, the number of ranks

k, m_max

int

As passed to the constructor; m_max is rounded up to 256 the same way

It returns the bytes the constructor allocates: 4 KiB of control plus 2 * M * K, with M the padded m_max, a send and a receive copy of the fp8 operand. The window does not depend on n, mode, chunks, wait or acquire. The method does not validate its arguments. Size the communicator with per_rank_vmm = 2 * window_bytes_for(...) + (256 << 20), as the overview and the tests do.

5. Calling the operator

c = op(a_fp8, b_preshuffled, a_scale, b_scale)    # torch.Tensor [M, N] bf16

Argument

Shape

Dtype

Description

a_fp8

[P*M, K/P], 2-D

torch.float8_e4m3fn, on the GPU

Every token, this rank’s K/P head columns. Rows [d*M, (d+1)*M) go to rank d. M = rows / P must be a multiple of 256 and at most m_max. Pass op.send_buffer(M) to skip the copy into the window; any other tensor is made contiguous and copied in.

b_preshuffled

[N, K]

torch.float8_e4m3fn, on the GPU

The replicated weight through mori.ops.gemm_ar.preshuffle_b, identical on every rank. Shuffle it once, at load time.

a_scale

[M] (any shape with M elements)

torch.float32, on the GPU

One scale per token of this rank, over all of K. See below.

b_scale

[N] (N elements)

torch.float32, on the GPU

One scale per output channel

The A scale covers all of K. Each rank quantises only its own head columns, but a token has one scale over every head, so every rank must quantise token t with the same scale[t]. Compute it before quantising: take each row’s absolute maximum over this rank’s heads ([P*M]), max-reduce it across ranks, and derive the scale from that. Quantise a[t] = attn_out[t] / scale[t] and pass this rank’s own tokens, a_scale = scale[rank*M:(rank+1)*M]. The scale never travels. A scale computed per head shard gives wrong results without an error. Section 10 shows the computation.

Returns C [M, N] bf16 on the current GPU: this rank’s tokens, [rank*M, (rank+1)*M) of the sequence. It is the operator’s buffer for that M (one per M), and the next call with the same M writes it. Read it before this rank’s next call, on the same stream or behind an event, or clone() it.

Ordering. The call is a collective: every rank makes the same calls, with the same M, in the same order. The work goes on the current stream: a copy into the window unless a_fp8 is send_buffer(M), the GEMM (which posts the bands) and the closing barrier. "split" runs a push and a barrier first. The send region may be refilled on the same stream as soon as a call is issued: the closing barrier holds every rank until every GEMM has finished reading. The first call at each M compiles its kernels. There are no call-time options: mode, chunks, wait and acquire are fixed at construction.

Padding. Pad per destination. Rank d’s block, rows [d*M, (d+1)*M), holds its real tokens first and zeros after them. Pad before computing the scales. Zero rows come back as zero rows of C; slice c[:tokens].

CUDA graphs. A captured call replays like a fresh call. Warm each M eagerly first, then refill the captured a_fp8 (or send_buffer(M)) and a_scale in place before each replay, the same number of times on every rank.

6. Other methods

padded_m()

op.padded_m(m: int) -> int rounds a per-rank token count up to a multiple of 256: the M to pass. It raises ValueError for m < 1. It does not check m_max; the call does.

send_buffer()

op.send_buffer(m: int) -> torch.Tensor returns the window’s send region for M = m, [P*m, K/P] float8_e4m3fn. Rows [d*m, (d+1)*m) go to rank d. Quantise the attention output straight into it and pass the same tensor as a_fp8: the call then skips its copy into the window. Views for different m share the same memory. Write into it on the stream of the previous call, or from a stream that waits on that one. m must be a multiple of 256 and at most m_max (ValueError).

self_test()

op.self_test() -> None is a collective check that the copy engines move bytes. It fills every send slab with rank + 1, runs the split exchange at m_max, and raises RuntimeError unless every slab received from a peer holds that peer’s value: without SDMA, every put returns and moves nothing. Call it once on every rank after construction, outside CUDA-graph capture (it synchronises and runs host barriers), and before writing into send_buffer, which it overwrites.

chunks_for() and wait_for()

op.chunks_for(m: int) -> int and op.wait_for(m: int) -> str return the band count and wait mode a call at M uses. These are the constructor’s chunks (rounded down to a divisor of M/256) and wait if given, otherwise default_chunks(M, P) and default_wait(M, P). Log them to record what a deployment runs with.

close() and the context manager

op.close() -> None releases the window and its memory. It is idempotent, and leaving a with A2aGemmOp(...) as op: block calls it. It does not synchronise: call torch.cuda.synchronize() first if a call may still be in flight. The device communicator the operator created stays with the communicator until that is destroyed. After close(), calling the operator, send_buffer() or self_test() raises RuntimeError. A constructor that fails releases the window before it raises.

Attributes

Attribute

Meaning

op.k_shard

K/P, the head columns each rank owns

op.m_max

m_max rounded up to a multiple of 256

op.window_bytes

The window’s size, window_bytes_for(P, k=k, m_max=m_max)

op.rank, op.world_size

This rank and P, from the communicator

op.n, op.k, op.mode, op.chunks, op.wait, op.acquire

The constructor’s arguments; chunks and wait stay None when defaulted

7. Shapes and limits

Constraint

Value

Error if violated

Ranks P

2 to 8

ValueError: world_size must be in [2, 8]

One node

lsa_size == P

RuntimeError: A2aGemmOp needs every rank in one LSA domain (one node)

SDMA queues

at least 1 per peer

RuntimeError: A2aGemmOp needs an SDMA queue per peer but the communicator has none: build mori with BUILD_CCO_SDMA=ON and set MORI_ENABLE_SDMA=1

n

multiple of 256

ValueError: n must be a multiple of 256

k

multiple of P * 128, at least 256

ValueError: K must be a multiple of world_size*128, or must be at least 256

m_max

at least 1

ValueError: m_max must be >= 1

Window span

2 * m_max * K below 2 GiB

ValueError: send+recv span … bytes; the A buffer descriptor takes 32-bit offsets

Per-rank VMM

holds the window

RuntimeError: ccoMemAlloc failed, with “slot exhausted … Increase perRankVmmSize” in the log

mode, wait, acquire

the values in section 4

ValueError: … is not one of …

chunks

None or 1 to 16

ValueError: chunks must be in [1, 16]

a_fp8 shape

2-D [P*M, K/P]

ValueError: a_fp8 must be [world*M, K/P]

M per call

multiple of 256, at most m_max

ValueError: M must be a multiple of 256; use padded_m(), or M exceeds the window’s m_max

b_preshuffled shape

(N, K)

ValueError: b_preshuffled must be (N, K)

fp8 operands

torch.float8_e4m3fn, not fnuz

ValueError: must be one of [‘torch.float8_e4m3fn’] …; the launch reinterprets it as raw bytes

Scales

float32, M and N elements

ValueError: must be float32, or must have M (N) elements

Device

every tensor on the GPU

ValueError: must be on a GPU

padded_m(m)

m >= 1

ValueError: m must be >= 1

After close()

—

RuntimeError: this A2aGemmOp has been closed

Not implemented: blockscale and mxfp8, N that is not a multiple of 256, a tile other than 256x256, and a per-shard A scale.

8. Choosing settings

default_chunks(m, world_size) and default_wait(m, world_size) pick the settings per M. They are fitted on the README’s two shapes:

P

Bands

Wait

2 to 7

M // 2048, at least 1: one band per 2048 rows

"segment"

8

1 up to M = 2048, then M // 1024

"segment" below M = 4096, "tile" from 4096

The band count is capped at 16 and rounded down to a divisor of M/256. At the six measured points the defaults pick the best cell of the sweep in section 9, except for the 70B shape at M = 4096. There they pick tile c4 (259.0 µs) against the best, tile c2 (258.6 µs).

When to override:

  • Other shapes or rank counts. Only P = 4 and P = 8 were measured; P = 2, 3, 5, 6 and 7 use the P = 4 rule. Sweep --chunks and --wait with the benchmark (section 11) and pin the winners with chunks= and wait=.

  • acquire="none" drops a cache invalidate after each wait. Every measurement here used it. It is correct while cco windows are uncached, the default; keep "inv" if CCO_UNCACHED_WINDOW=0 is set.

  • mode="split" is the baseline and a fallback. It pushes in chunks_for(M) bands; the README’s split-sdma column pushes one.

The operator reads no environment variables itself. These affect it:

Variable

Effect

MORI_ENABLE_SDMA

Must be 1 when the communicator is created; without it there are no SDMA queues and the constructor raises

CCO_UNCACHED_WINDOW

Unset (default): cco windows are uncached. 0: cached memory, where acquire="none" is not safe

MORI_SOCKET_IFNAME

Network interface for the communicator’s unique-id rendezvous; the tests and benchmarks use lo

9. Performance

fp8 ptpc in, bf16 out, 256x256 tile. MI355X, one node, 2026-10-28, FlyDSL 0.4.0, torch 2.12.0+rocm7.14.0. Every case ran in one process on one communicator and replayed its CUDA graph back to back, with an event between launches, for at least 3 s. Each reports the median launch gap, maximum over ranks; three rounds were interleaved (spread 0.0-2.0%). Every case was validated against an fp32 reference at the fp8 floor (relL2 1.66e-3), with acquire="none". gemm-only is the GEMM over an already exchanged operand; split-rccl is all_to_all_single, then the same GEMM; split-sdma is push (one band), barrier, the same GEMM, barrier. Times in µs:

Shape

M per rank

gemm-only

split-rccl

split-sdma

best fused

vs split-sdma

vs RCCL

MiniMax-M3 o-proj, P=4, N=K=6144

2048

63.0

144.0

132.9

121.7 segment c1

-8.5%

-15.5%

4096

125.1

264.9

244.0

184.1 segment c2

-24.5%

-30.5%

8192

208.6

479.0

438.7

300.2 segment c4

-31.6%

-37.3%

Llama-3.1-70B o-proj, P=8, N=K=8192

2048

94.3

156.2

145.8

134.4 segment c1

-7.8%

-14.0%

4096

182.6

294.6

269.6

258.6 tile c2

-4.1%

-12.2%

8192

356.4

552.5

506.4

435.2 tile c8

-14.0%

-21.2%

Every fused configuration of the same run (cN is N bands):

Shape

M per rank

wait

c1

c2

c4

c8

M3 P=4

2048

segment

121.7

128.2

136.8

157.5

2048

tile

128.8

136.6

144.6

165.5

4096

segment

234.0

184.1

190.8

209.4

4096

tile

240.7

188.1

198.8

217.0

8192

segment

424.0

356.1

300.2

317.6

8192

tile

434.7

363.7

307.6

326.2

70B P=8

2048

segment

134.4

139.0

149.0

176.0

2048

tile

141.8

141.8

151.6

179.0

4096

segment

262.4

259.7

261.4

263.0

4096

tile

264.0

258.6

259.0

259.7

8192

segment

515.2

499.8

466.0

448.3

8192

tile

506.9

482.9

452.8

435.2

Through A2aGemmOp with its defaults (acquire="none", graph replay, input written into send_buffer), the 70B shape at P=8 measured 136.6 / 262.0 / 430.9 µs fused and 149.7 / 290.1 / 557.4 µs split, at M = 2048 / 4096 / 8192 per rank.

  • Fusing wins in all six cells, by 4-32% against split-sdma and 12-37% against RCCL, most where the transfer is the larger half: at P=4 the exchange (split-sdma minus gemm-only) is 70-230 µs against a 63-209 µs GEMM.

  • The bands are the mechanism. With one band a remote segment waits for a peer’s whole slab; more bands let a wave start after a fraction of the transfer, and smaller bands lose bandwidth per put. At P=4 the best count rises with M: 1, 2, 4.

  • segment is best or within 0.5% in five of six cells, and wins whenever the tiles fit in one wave (M = 2048). tile is ahead only at P=8 with M >= 4096, by 0.4-3%.

  • At P=8 the fused path still trails the GEMM alone by 40-80 µs. The first wave of tiles needs half of the transferred bytes, and they arrive at the link’s rate. Closing that gap needs the bands posted before the GEMM starts, from attention’s epilogue, which is not done yet.

10. Complete example

# o_proj.py: MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=8 o_proj.py
import os

import torch
import torch.distributed as dist

from mori.cco import Communicator, UniqueId
from mori.ops.a2a_gemm import A2aGemmOp
from mori.ops.gemm_ar import preshuffle_b

N = K = 8192                       # Llama-3.1-70B o-projection
M_MAX = 8192                       # most tokens per rank this deployment will see

torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo,cuda:nccl")
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)

# Stand-in for the checkpoint's fp8 o_proj weight and per-channel scale.
g = torch.Generator(device="cuda").manual_seed(0)       # the same on every rank
w_o = (torch.randn(N, K, generator=g, device="cuda") / 8).to(torch.float8_e4m3fn)
b_scale = torch.rand(N, generator=g, device="cuda") * 0.01 + 0.01      # [N] fp32
b = preshuffle_b(w_o)

tokens = 4000                                          # real tokens per rank
vmm = 2 * A2aGemmOp.window_bytes_for(world, k=K, m_max=M_MAX) + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
    with A2aGemmOp(comm, n=N, k=K, m_max=M_MAX) as op:
        op.self_test()
        m = op.padded_m(tokens)                        # 4096
        a = op.send_buffer(m)                          # [P*m, K/P] fp8, in the window
        a_scale = torch.empty(m, device="cuda")        # [m] fp32

        def prepare():
            # Stand-in for Ulysses attention: every token, this rank's heads.
            # Rank d's tokens are rows [d*m, d*m + tokens); the padding is zero.
            attn = torch.randn(world, m, op.k_shard, device="cuda", dtype=torch.bfloat16)
            attn[:, tokens:] = 0
            attn = attn.view(world * m, op.k_shard)
            amax = attn.abs().amax(dim=1).float()      # [P*m], over this rank's heads
            dist.all_reduce(amax, op=dist.ReduceOp.MAX)  # over every rank's heads
            fp8_max = torch.finfo(torch.float8_e4m3fn).max
            scale = (amax / fp8_max).clamp_min(1e-12)  # the same on every rank
            quantize_rows_into(attn, scale, out=a)     # your quantiser
            a_scale.copy_(scale[rank * m:(rank + 1) * m])

        prepare()
        side = torch.cuda.Stream()                     # eager call on a side stream: it
        side.wait_stream(torch.cuda.current_stream())  # compiles this M and warms it
        with torch.cuda.stream(side):
            c = op(a, b, a_scale, b_scale)             # [m, N] bf16
            y = c[:tokens].clone()
        torch.cuda.current_stream().wait_stream(side)

        # Capture, then refill and replay. thread_local: RCCL's watchdog thread
        # may query events while this one captures.
        graph = torch.cuda.CUDAGraph()
        with torch.cuda.graph(graph, capture_error_mode="thread_local"):
            c = op(a, b, a_scale, b_scale)
        for _ in range(3):
            prepare()
            graph.replay()
            y = c[:tokens].clone()                     # before the next call writes c
        torch.cuda.synchronize()                       # nothing in flight at close()
dist.destroy_process_group()

quantize_rows_into stands for your quantiser; MORI ships none. It must write a[t] = attn[t] / scale[t] as torch.float8_e4m3fn for all P*m rows, with the scale every rank shares (unfused: a.copy_((attn.float() / scale[:, None]).to(torch.float8_e4m3fn))). a_scale is this rank’s m entries of that scale, fp32.

11. Testing and benchmarking

export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest -q tests/python/cco/test_a2a_gemm.py tests/python/cco/test_a2a_gemm_op.py
  • test_a2a_gemm.py checks the window layout on the CPU. On 2 to 8 GPUs it runs the benchmark over every mode, 1 and 2 bands, both waits and both acquire values with 8 back-to-back calls, then signal counters that wrap past 2^32 and band counts that change between calls.

  • test_a2a_gemm_op.py needs 8 GPUs: self_test(), "fused" and "split" over several M on one instance (with send_buffer and graph replays), back-to-back calls with one rank 20 ms late, and the constructor’s refusals, with SDMA on and off.

Allow about 10 s between back-to-back 8-rank runs so the SDMA queues are released; test_a2a_gemm_op.py retries once when it hits this.

bench_a2a_gemm.py times gemm-only, split-rccl, split-sdma and fused-sdma, driving the kernel builders directly rather than A2aGemmOp. The MiniMax-M3 rows above came from this command; for the 70B rows, use 8 GPUs with --nproc_per_node=8 -n 8192 -k 8192:

HIP_VISIBLE_DEVICES=0,1,2,3 MORI_SOCKET_IFNAME=lo MORI_ENABLE_SDMA=1 \
  torchrun --standalone --nproc_per_node=4 \
  benchmark/cco/flydsl/a2a_gemm/bench_a2a_gemm.py -m 2048 4096 8192 \
  -n 6144 -k 6144 --chunks 1 2 4 8 --wait segment tile --acquire none \
  --rounds 3 --out m3.jsonl

Flag

Default

Meaning

--modes

all four

Any of gemm-only, split-rccl, split-sdma, fused-sdma

-m, -n, -k

2048, 6144, 6144

Tokens per rank after the exchange (several allowed), N and K

--chunks, --wait

1 2 4, segment

Band counts and waits for fused-sdma; a count is skipped unless M is a multiple of 256 * count. --wait none (with --no-post) is a diagnostic and is not validated.

--acquire

inv

inv or none

--rounds

1

Rounds over every case, alternating forward and reverse order

--warmup, --iters, --min-seconds

20, 100, 3.0

Warm-up launches; the least number of launch gaps and seconds timed

--back-to-back

0

Adds that many calls with no host sync, one rank held before each, and validates every one

--tolerance, --out

3e-3, none

The largest relL2 that passes; a JSONL file rank 0 appends one record per case to

Each case also prints a RESULT_JSON line, and the script exits 1 if any case fails validation. timeline_a2a_gemm.py, in the same directory, records when each band lands and when each tile starts and ends; its docstring has the command.

12. Troubleshooting

Error or symptom

Cause

What to do

ModuleNotFoundError: No module named 'flydsl' on importing A2aGemmOp

FlyDSL is not installed

pip install 'amd_mori[flydsl]'

RuntimeError: A2aGemmOp needs an SDMA queue per peer ...

MORI built with BUILD_CCO_SDMA=OFF, MORI_ENABLE_SDMA not set when the communicator was created, or no peer reports SDMA. cco logs “no SDMA queues will be created”.

Rebuild with BUILD_CCO_SDMA=ON; export MORI_ENABLE_SDMA=1 before Communicator.init

RuntimeError: a2a_gemm self-test failed: recv slab from rank s holds v, expected s+1

The copy engines move no bytes

As for the SDMA error above

RuntimeError: A2aGemmOp needs every rank in one LSA domain (one node)

The ranks span nodes

Run every rank on one node

RuntimeError: ccoMemAlloc failed, “slot exhausted” in the log

per_rank_vmm is too small for the window

Size it from window_bytes_for with the same k and m_max

ValueError: M=... exceeds the window's m_max=...

M above the window

Construct with a larger m_max; the window cannot grow

ValueError: a_fp8 must be [world*M, ...]

a_fp8 is [M, K], or has the wrong width

Pass every token, with this rank’s K/P columns

ValueError: a_scale must have M elements, got P*M

The whole [P*M] scale was passed

Pass scale[rank*M:(rank+1)*M]

A call never returns

A rank skipped a call, used another M, or was built with another mode, or (fused) a sender splits its slab into fewer chunks than its receiver, so a band or a barrier never arrives

Same arguments and the same calls on every rank

Wrong results, no error

A per-shard A scale; the same scale slice on every rank; (fused) a sender with more chunks than its receiver, whose band waits then pass before the rows have landed; a weight not through preshuffle_b or not identical on every rank; acquire="none" with CCO_UNCACHED_WINDOW=0; c read after the next call at that M; send_buffer written from a stream not ordered after the previous call

See 5. Calling the operator

The first call at a new M is slow

It compiles that M’s kernels

Warm every M before timing or capturing

Further reading