GEMM + all-to-all
GemmA2AOp (mori.ops.gemm_a2a) is an fp8 GEMM with a column-sharded
all-to-all fused into it. Every rank computes C = A @ B.T for its own rows
and receives, from every rank, the block of columns it owns. This is the QKV
projection under Ulysses sequence parallelism. The GEMM epilogue stages C per
destination, and the SDMA copy engines push each chunk as soon as its tiles are
done, while the GEMM computes the rest.
Table of Contents
Quick Reference
The snippet assumes the setup in
What every operator has in common:
FlyDSL and SDMA, a communicator whose VMM reservation is sized with
window_bytes_for, and one self_test() per operator.
from mori.ops.gemm_a2a import GemmA2AOp
from mori.ops.gemm_ar import preshuffle_b
# comm: a mori.cco.Communicator with per_rank_vmm sized from
# GemmA2AOp.window_bytes_for(world, n=10240, m_max=16384), see section 10
with GemmA2AOp(comm, n=10240, k=8192, m_max=16384, quant="ptpc") as op:
op.self_test() # collective, once
b_shuffled = preshuffle_b(w_qkv) # [N, K], same on every rank
m = op.padded_m(x.shape[0])
a_fp8, a_scale = quantize_per_token(op.pad_rows(x, m)) # your quantiser
qkv = op(a_fp8, b_shuffled, a_scale, b_scale) # [P*m, N/P] bf16
What |
Import |
|---|---|
Operator |
|
Weight preshuffle (also re-exported by |
|
mxfp8 A-scale layout |
|
Communicator |
|
1. What it computes
P is the number of ranks, M the rows each rank passes and s = N/P the
columns each rank receives. Rank r passes its own A_r [M, K] and the same
B [N, K] as every other rank, and computes C_r = A_r @ B.T ([M, N], never
as one tensor). Column block d of C_r goes to rank d. Rank d gets
[P*M, s] bf16, source rank r at rows [r*M, (r+1)*M), its own block
included. With ptpc, rank d gets, up to rounding:
c = [(A[r].float() @ B.float().T) * a_scale[r][:, None] * b_scale[None, :] for r in range(P)]
out_d = torch.cat([c_r[:, d * s:(d + 1) * s] for c_r in c]).to(torch.bfloat16) # [P*M, s]
Under Ulysses, K is the hidden size, N the QKV width, s one rank’s head
shard and M = S/P, so a rank gets every token for its own heads. Output
column i comes from weight row i, so order the weight’s rows per
destination: rank j’s Q, K and V heads in rows [j*s, (j+1)*s). For
Llama-3.1-70B at P=8, s = 1280 is 8 q heads, 1 k head and 1 v head of 128:
one whole GQA group per rank. A and B are torch.float8_e4m3fn; the output
is bf16.
2. How it works
The GEMM is gemm_ar’s 8-wave fp8 kernel; only where C goes is different.
Staging. Tiles are 256 columns wide and
sis a whole number of tiles, so each tile belongs to one destination. The epilogue stores it into this rank’s staging slab for that destination,[P][M][s]in the window, so a destination’s bytes are one contiguous range: a copy engine cannot gather. This rank’s own columns go straight into its receive region. Blocks visit the destinations round-robin from a rank-dependent start, so every link has bytes to move early.Chunks. Each destination’s slab is cut along M into
chunksruns of row tiles. A counter per (destination, chunk) finds the block that finishes a chunk’s last outstanding tile; that block posts one SDMA put of the chunk, and the copy engine moves it while the GEMM computes later tiles.Close. A one-block kernel waits for this rank’s puts to drain and meets every peer at a barrier. The receive region then holds all
Pblocks.
mode="split" runs the same GEMM into the same slabs, then one kernel that
posts one put per destination, drains them and meets the peers. Fusion cannot
hide the wait for the first chunk, the last chunk’s transfer, or differences in
GEMM speed between ranks; the more GEMM follows the first chunk, the more of
the transfer it hides, so the gain grows with M. There is no LSA (peer-store)
transport: it lost to the copy engines on this operator and was removed.
Receive reuse. Each call posts a credit to every peer when it starts, and a
put to rank d waits for d’s credit for the same call. A rank that is a call
ahead therefore cannot overwrite a result a slower rank is still reading,
provided the reader is ordered before the slower rank’s next call (section 5).
The window holds a control region of a few KiB (barrier flags, one set of
completion counters per distinct M, submit locks, credits), the staging slabs
[P, m_max, s] and the receive region [P*m_max, s], both bf16: about
4 * m_max * N bytes, 640 MiB at N=10240, m_max=16384.
3. Requirements
The shared setup is in What every operator has in common. For this operator:
Requirement |
Detail |
|---|---|
GPU |
gfx950 (MI355X). The GEMM is |
FlyDSL |
|
SDMA |
MORI built with |
Ranks |
P from 2 to 8, all in one node (one LSA domain), one process per GPU. Set the device before constructing: the operator reads its CU count to pick the tile. |
Communicator |
A |
4. Constructor
GemmA2AOp(comm, *, n, k, m_max, quant="ptpc", mode="fused",
sdma_queues=1, max_shapes=8, chunks=None, war_credit=True)
Collective: every rank passes the same arguments. It allocates and zeroes the
window, creates a dev-comm and ends in a barrier; kernels compile on the first
call at each M. The weight is a call argument, so layers that share (n, k)
can share one instance.
Name |
Type |
Default |
Description |
|---|---|---|---|
|
|
(required) |
Every rank in one node. Its VMM reservation must hold |
|
|
(required) |
N, the full output width (the QKV width). A multiple of |
|
|
(required) |
K, the hidden size. A multiple of 128, at least 256. |
|
|
(required) |
The largest per-rank M any call passes, at least 1. Rounded up to the M granule (128, or 256 for mxfp8). Sizes the window, which cannot grow. |
|
|
|
|
|
|
|
|
|
|
|
SDMA queues per peer, at least 1. The communicator’s first dev-comm fixes the count, and the constructor raises if it has fewer. Leave it at 1 (section 8). |
|
|
|
How many distinct padded M values the instance serves, at least 1. Each takes a counter set in the window; |
|
|
|
Puts per destination per call, at least 1. |
|
|
|
The receive-reuse credit. |
window_bytes_for()
GemmA2AOp.window_bytes_for(world_size, *, n, m_max, quant="ptpc", max_shapes=8, chunks=None) -> int
Static. Returns the exact size the constructor allocates (op.window_bytes), so
the communicator can be sized before the operator exists. world_size is P
(the communicator’s nranks); pass n, m_max, quant, max_shapes and
chunks as you pass them to the constructor, with the same defaults. quant
sets the M granule m_max is padded to, max_shapes the number of counter
sets and chunks their capacity, max(8, chunks). k, mode and the tile do
not enter it. A bad world_size or n raises the constructor’s ValueError.
5. Calling the operator
out = op(a_fp8, b_preshuffled, a_scale, b_scale) # [P*M, N/P] bf16
A call is collective (every rank calls, with the same M, in the same order) and
is issued on the current stream. There are no optional arguments: no out=
and no stream argument.
Argument |
Shape |
dtype |
Notes |
|---|---|---|---|
|
|
|
This rank’s rows. M is a multiple of |
|
|
|
The whole weight through |
|
ptpc: |
ptpc: |
ptpc: one dequantisation scale per row. mxfp8: |
|
ptpc: |
ptpc: |
ptpc: one scale per output channel. mxfp8: the |
All four must be on the GPU. FNUZ fp8 (float8_e4m3fnuz) is rejected.
The return value is a [P*M, N/P] bf16 view of the operator’s window, rows
[r*M, (r+1)*M) from rank r. Every call at the same M returns the same
tensor, and this rank’s next call, at any M, overwrites it. Whatever reads it
must be ordered before that call on the device: the same stream, or an event
the next call’s stream waits on. Clone it to keep it longer. If you padded m
rows to M, each source’s block ends in M - m padding rows, zero when your
quantiser maps a zero row to zeros with a finite scale. Drop them per source:
out.view(P, M, -1)[:, :m].
The first call at each M compiles that M’s kernels (FlyDSL JIT): warm every M before timing or capturing a CUDA graph. A replayed call behaves like a fresh one, because the call counter behind the credit and the barrier lives on the device.
6. Other methods
padded_m()
op.padded_m(m: int) -> int rounds the logical row count up to op.m_granule
(128, or 256 for mxfp8). Call it on each call’s row count, then pad_rows().
ValueError if m < 1. It does not check m_max.
pad_rows()
op.pad_rows(x: torch.Tensor, m_pad: int) -> torch.Tensor zero-extends a 2-D
[m, K] tensor of any dtype to [m_pad, K], in a buffer the operator reuses:
the next pad_rows overwrites the returned view. Pad before quantising, so the
scales cover the padded rows. ValueError if x is not 2-D or m_pad < m.
self_test()
op.self_test() -> None is collective: call it on every rank once after
construction, before real calls. It writes a known value into every staging
slab, runs the split push at m_max, synchronises and checks every remote
block. A MORI built without BUILD_CCO_SDMA, or run without
MORI_ENABLE_SDMA=1, moves nothing; this then raises RuntimeError instead of
letting calls return stale data. It overwrites the staging and receive regions
and registers m_max as one of the max_shapes M values.
block_m_for() and chunks_for()
op.block_m_for(m: int) -> int is the tile’s M (128 or 256) and
op.chunks_for(m: int) -> int the puts per destination that a call at padded
m uses. Host-side only; use them to log or check what a shape runs at.
close() and the context manager
op.close() -> None releases the window and its memory; with GemmA2AOp(...) as op: calls it on exit. It is idempotent. Synchronise first if a call may
still be in flight. Afterwards calls and self_test() raise RuntimeError,
and tensors the operator returned point at released memory. The dev-comm it
created stays with the communicator until the communicator is destroyed. A
constructor that fails releases what it had allocated.
Attributes
op.m_max is m_max rounded up to the granule, op.m_granule is 128 or 256,
op.shard_n is N/P, op.window_bytes equals window_bytes_for(...), and
op.rank and op.world_size come from the communicator. The other
constructor arguments are kept under their own names.
7. Shapes and limits
Constraint |
Value |
Error if violated |
|---|---|---|
Ranks |
P from 2 to 8 |
|
One node |
every rank in one LSA domain |
|
N |
a multiple of |
|
K |
a multiple of 128, at least 256 |
|
|
|
|
|
at least 1 ( |
|
SDMA queues on the communicator |
at least |
|
VMM reservation |
holds the window |
|
M per call |
a multiple of 128 (256 for mxfp8), at most |
|
Distinct M per instance |
at most |
|
|
|
|
Scales |
dtype and element count as in section 5, on the GPU |
|
Instance open |
not closed |
|
At P=8 the QKV widths of Llama-3.1-8B, 70B and 405B, Qwen2.5-72B and Qwen3-32B fit. Qwen3-235B-A22B (N=9216) does not: it has fewer KV heads than ranks, so Ulysses at P=8 would have to replicate KV.
8. Choosing settings
Setting |
What the default does |
When to change it |
|---|---|---|
Tile (not a parameter; |
mxfp8: 256x256, always (the packed A scale needs it). ptpc: 256x256 when M is a multiple of 256 and that grid is at least two waves, |
Only the benchmark’s |
|
The largest count up to 8 that divides the row tiles and keeps every put at least 1 MiB, |
With ptpc at P=8 the default is the measured best except 70B M=8192, where 4 and 8 are within 1%; with mxfp8, 70B did best with 4 at M=4096 and 8192, where the default is 8. At P=4, M=2048, |
|
Puts from the epilogue. |
Keep it: it beat |
|
fp32 per-token and per-channel scales. |
Use the format your checkpoint is in. At the same tile, mxfp8’s GEMM is 13-22% slower than ptpc’s and its best fused path 10-21% slower: its scale loads stall the tile prefetch. |
|
One queue per (source, destination) pair, so every peer already has its own. |
Leave it. Asking for P makes P per peer and uses one; in a process that already holds SDMA engines, queue creation then fails ( |
|
The window is about |
Set |
The operator reads no environment variables itself. MORI needs
MORI_ENABLE_SDMA=1, read once when its context is created (with the
communicator), and BUILD_CCO_SDMA=ON at build time (the default; OFF builds
a library whose puts move nothing). MORI_SOCKET_IFNAME picks the interface for
the communicator’s rendezvous; the tests set lo
(CCO guide).
9. Performance
Conditions. 8x MI355X (gfx950), one otherwise idle node, 2026-10-28; fp8
e4m3 in, bf16 out; FlyDSL 0.4.0, torch 2.12.0+rocm7.14.0. Every case ran in one
process on one communicator (sweep_inproc.py), through the benchmark, which
builds the kernels the operator builds. A case replays its CUDA graph back to
back, 50 launches to a batch with an event between each, for at least 3 s (8 s
in the last round) and 100 launches. The number is the median launch-to-launch
gap, maximum over ranks. Three interleaved rounds, spread 0.0-3.3%; every
sample validated against a host reference on every source rank.
split SDMA is mode="split". split RCCL is the same GEMM, then
all_to_all_single. best fused is mode="fused" at the better of 4 and 8
chunks (in parentheses). Times in µs; negative is faster.
Shape (fp8 ptpc, P=8) |
M (S) |
Tile |
GEMM only |
split SDMA |
split RCCL |
best fused |
vs split SDMA |
vs RCCL |
|---|---|---|---|---|---|---|---|---|
Llama-3.1-70B / Qwen2.5-72B, N=10240 K=8192 |
2048 (16k) |
128x256 |
151 |
244 |
344 |
203 (c4) |
-16.6% |
-40.9% |
4096 (32k) |
256x256 |
257 |
441 |
513 |
328 (c8) |
-25.5% |
-36.0% |
|
8192 (64k) |
256x256 |
451 |
813 |
930 |
578 (c4) |
-28.8% |
-37.8% |
|
16384 (128k) |
256x256 |
918 |
1619 |
1824 |
1138 (c8) |
-29.7% |
-37.6% |
|
Llama-3.1-405B, N=18432 K=16384 |
2048 (16k) |
256x256 |
502 |
681 |
738 |
568 (c8) |
-16.6% |
-23.1% |
4096 (32k) |
256x256 |
899 |
1204 |
1303 |
989 (c8) |
-17.9% |
-24.1% |
|
8192 (64k) |
256x256 |
1699 |
2280 |
2457 |
1858 (c8) |
-18.5% |
-24.4% |
|
16384 (128k) |
256x256 |
3383 |
4694 |
4998 |
3678 (c8) |
-21.6% |
-26.4% |
mxfp8 (256x256, same node and run, two rounds, spread 0.0-2.3%) gains 11-24% over split SDMA at the same shapes; the README has its table.
Against aiter #6689’s fused sp_qkv, MiniMax-M3 QKV, P=4. K=6144; dense
sends Q, K, V per destination (N=9216), indexer adds IQ and IK (N=10240; mori
replicates IK’s weight rows per destination, 3.9% more GEMM work). aiter
474128f, r13 kernels, through SpQkvLarge.run; mori fp8 ptpc. GPUs 0-3 of one
8x MI355X node, ROCm 7.14 CI image, FlyDSL 0.4.0. Both sides: graph replay
back to back with an event between launches, median gap, maximum over ranks,
three interleaved rounds (spread at most 0.4%), all validated at relL2
1.66e-3. 128 c8: 128x256 tile, 8 chunks.
Case |
M |
aiter r13 |
|
vs aiter |
best arm |
vs aiter |
|---|---|---|---|---|---|---|
dense |
2048 |
247.0 |
242.4 |
-1.8% |
231.8 |
-6.2% |
dense |
4096 |
439.3 |
431.0 |
-1.9% |
404.4 |
-7.9% |
dense |
8192 |
778.1 |
751.7 |
-3.4% |
747.2 |
-4.0% |
indexer |
2048 |
289.6 |
259.5 |
-10.4% |
248.6 |
-14.2% |
indexer |
4096 |
475.2 |
467.4 |
-1.6% |
438.8 |
-7.7% |
indexer |
8192 |
870.6 |
825.4 |
-5.2% |
814.9 |
-6.4% |
How to read them:
Fusion wins in every measured cell, and its margin over split SDMA grows with M (70B: 16.6% to 29.7%). Nothing below M=2048 was measured.
At large M the transfer is nearly hidden, which bounds the gain: at 405B M=16384 the exposed communication drops from 1372 to 176 µs (-87%), but the GEMM is 95% of the fused total, so end to end the gain is 21.6%.
split RCCLruns the same GEMM over the same bytes in the same layout, so its column compares transports only. RCCL was last in every cell.At P=4 the defaults are 1.6-10.4% ahead of aiter but leave up to 6.6% against the best arm (M=4096): the tile rule was fitted at P=8.
10. Complete example
quantize_per_token stands for your own quantiser; MORI ships none. For ptpc
it returns a_fp8 [m, K] float8_e4m3fn and a_scale [m] fp32, with
x[i] ~= a_fp8[i].float() * a_scale[i]. For mxfp8 it returns the [m, K/32]
ue8m0 bytes, which go through preshuffle_a_scale.
# qkv_a2a.py -- MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=8 qkv_a2a.py
import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_a2a import GemmA2AOp
from mori.ops.gemm_ar import preshuffle_b
N, K, M_MAX = 10240, 8192, 16384 # Llama-3.1-70B QKV; M_MAX is the largest S/P served
TOKENS = 4000 # this rank's S/P here; padded to 4096
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo")
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
vmm = 2 * GemmA2AOp.window_bytes_for(world, n=N, m_max=M_MAX, quant="ptpc") + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with GemmA2AOp(comm, n=N, k=K, m_max=M_MAX, quant="ptpc") as op:
op.self_test() # collective; raises if SDMA moves nothing
g = torch.Generator(device="cuda").manual_seed(0) # the same weight on every rank
w = (torch.randn(N, K, generator=g, device="cuda") / 8).to(torch.float8_e4m3fn)
w_scale = torch.full((N,), 0.01, device="cuda") # [N] fp32, per output channel
b = preshuffle_b(w)
m = op.padded_m(TOKENS)
x = torch.randn(TOKENS, K, device="cuda", dtype=torch.bfloat16)
a_fp8, a_scale = quantize_per_token(op.pad_rows(x, m)) # your quantiser
out = op(a_fp8, b, a_scale, w_scale) # the first call at this M compiles it
qkv = out.view(world, m, -1)[:, :TOKENS] # [P, TOKENS, N/P], padding dropped
# CUDA graph: warm on a side stream, capture, then refill the static
# inputs and replay. A replay is a call, so every rank replays.
static_a, static_s = a_fp8.clone(), a_scale.clone()
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
op(static_a, b, static_s, w_scale)
torch.cuda.current_stream().wait_stream(side)
torch.cuda.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
out = op(static_a, b, static_s, w_scale)
for _ in range(3):
x = torch.randn(TOKENS, K, device="cuda", dtype=torch.bfloat16)
a_new, s_new = quantize_per_token(op.pad_rows(x, m)) # your quantiser
static_a.copy_(a_new)
static_s.copy_(s_new)
graph.replay()
qkv = out.view(world, m, -1)[:, :TOKENS].clone() # same stream, before the next call
torch.cuda.synchronize() # nothing in flight before close()
dist.destroy_process_group()
11. Testing and benchmarking
From the repository root:
export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest tests/python/cco/test_gemm_a2a.py # window layout and shape rules, on the CPU
pytest tests/python/cco/test_gemm_a2a_op.py # GemmA2AOp on 8 ranks
test_gemm_a2a_op.py needs 8 GPUs and FlyDSL, and skips without them. It runs
self_test(), both modes and both quantisations at two M values and through a
CUDA graph, the receive-reuse race (with war_credit=False as the negative
control) and the constructor’s refusals, each through
tests/python/cco/gemm_a2a_op_worker.py under torch.distributed.run. CI runs
both files from /tmp with absolute paths, so the installed mori is tested.
benchmark/cco/flydsl/gemm_a2a/bench_gemm_a2a.py calls the kernel builders,
not GemmA2AOp. Its --chunks defaults to 1, so pass the count you want;
--block-m 0 (the default) applies the operator’s tile rule. --mode is
gemm-only, gemm-staging, split-sdma, split-rccl, fused-sdma or
gemm-to-window; other flags include --quant, --chunk-plan, --timing {sync,b2b,stream}, --timeline PATH and --no-war-credit. The transfer modes
check what arrived from every source against a host reference; each run prints
one RESULT_JSON line. sweep_inproc.py runs a whole table in one process and
skips rows its --out already holds, so a stopped sweep resumes.
# one cell
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
python -m torch.distributed.run --standalone --nproc_per_node=8 \
benchmark/cco/flydsl/gemm_a2a/bench_gemm_a2a.py \
--mode fused-sdma --chunks 8 -m 16384 -n 18432 -k 16384 \
--warmup 30 --iters 21 --phase-split
# the ptpc tables (mxfp8: --quant mxfp8 --rounds 2)
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
python -m torch.distributed.run --standalone --nproc_per_node=8 \
benchmark/cco/flydsl/gemm_a2a/sweep_inproc.py --suite models --quant ptpc \
--alt-tile --rounds 3 --out models.jsonl
# the MiniMax-M3 comparison, on four GPUs
HIP_VISIBLE_DEVICES=0,1,2,3 MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
python -m torch.distributed.run --standalone --nproc_per_node=4 \
benchmark/cco/flydsl/gemm_a2a/sweep_inproc.py --suite minimax --rounds 3 \
--out minimax.jsonl
Keep the node idle and interleave rounds: a neighbour on the box was worth 9% on a pure GEMM and changed which chunk count won.
12. Troubleshooting
Symptom |
Cause and fix |
|---|---|
|
FlyDSL is not installed: |
|
MORI built without |
|
The puts move nothing; same causes. Without this check the operator would return this rank’s block plus stale data. |
|
The ranks span nodes. The operator is single-node. |
|
|
|
The kernel reads OCP |
A launch fails in |
The previous run’s SDMA queues are not reclaimed yet. Wait for its processes to exit and retry. Keep |
A call hangs, or rows land in the wrong place |
The ranks did not make the same calls, with the same M, in the same order. Every call waits on every peer. |
The result changes after you read it |
It was read after this rank’s next call, or on a stream not ordered before it. Read it on the same stream, or clone it. |
Wrong values, no error |
B differs between ranks; a scale is in the wrong layout (mxfp8 |
Any other |
An argument check; 7. Shapes and limits lists them. |
Further reading
The
gemm_a2aREADME, next to the code: the full design, every measurement (the mxfp8 table, the tile table, the removed LSA transports), the negative results and what was not measured.MORI-FUSED overview: what the six operators share, and Performance at a glance.
All-to-all + GEMM: the reverse operator, for the o-projection after Ulysses attention.