All-to-all + GEMM
A2aGemmOp (mori.ops.a2a_gemm) fuses the all-to-all at the end of Ulysses
attention into the fp8 GEMM of the o-projection. Every rank passes its head
shard for every token, [P*M, K/P], and the same weight B [N, K], and gets
back C [M, N] in bf16: its own M tokens over all of K. Block 0 of the GEMM
posts every band of rows to every peer over the SDMA copy engines. The GEMM
starts on this rank’s own K segment and reads each remote segment as soon as
its band has landed. This is the reverse of GemmA2AOp.
Table of Contents
Quick Reference
The snippet assumes the setup in
What every operator has in common:
FlyDSL and SDMA, a communicator whose VMM reservation is sized with
window_bytes_for, and one self_test() per operator.
from mori.cco import Communicator, UniqueId
from mori.ops.a2a_gemm import A2aGemmOp
from mori.ops.gemm_ar import preshuffle_b
# rank, world and uid (a mori.cco.UniqueId): see the complete example
vmm = 2 * A2aGemmOp.window_bytes_for(world, k=K, m_max=max_tokens_per_rank) + (256 << 20)
with Communicator.init(world, rank, uid, per_rank_vmm=vmm) as comm:
with A2aGemmOp(comm, n=N, k=K, m_max=max_tokens_per_rank) as op:
op.self_test() # collective; raises if no bytes move
b = preshuffle_b(w_o) # [N, K] fp8, the same on every rank
m = op.padded_m(tokens_per_rank)
a = op.send_buffer(m) # [P*m, K/P] fp8, in the window
quantize_attention_output_into(a) # your quantiser
c = op(a, b, a_scale, b_scale) # [m, N] bf16; a_scale [m], b_scale [N]
What |
Import |
|---|---|
Operator |
|
Weight layout |
|
Communicator |
|
Default bands and wait per M |
|
Limits ( |
|
Window layout and kernel builders, which the benchmark drives |
|
1. What it computes
Ulysses sequence parallelism over P ranks ends attention with every rank
holding all P*M tokens for its own K/P columns of heads. The o-projection
needs this rank’s M tokens over all of K, times the replicated weight
B [N, K]. Rank d computes
C_d = concat_s(A_s[d*M:(d+1)*M]) @ B.T # [M, N]
C_d[i, j] = a_scale[i] * b_scale[j] * sum_k A[i, k] * B[j, k]
A_s is source rank s’s input. The concatenation is along K: source s
owns columns [s*K/P, (s+1)*K/P), and A is the gathered [M, K] operand.
Unfused, this is all_to_all_single followed by the GEMM.
Per rank |
Shape |
Dtype |
Content |
|---|---|---|---|
|
|
|
Every token, this rank’s head columns. Rows |
|
|
|
The o-projection weight through |
|
|
|
One scale per token of this rank, over all of K |
|
|
|
One scale per output channel |
|
|
|
This rank’s tokens, rows |
fp8 ptpc is the only precision. The README measures the MiniMax-M3 o-projection at Ulysses SP4 and the Llama-3.1-70B o-projection at P=8.
2. How it works
The operator’s window keeps what this rank sends and what it receives side by side, in the same per-source layout:
[ control | send: [P][M][K/P] fp8 | recv: [P][M][K/P] fp8 ]
send[d], the rows bound for rank d, is one contiguous slab, and
send_buffer(M) is the whole send region as [P*M, K/P]. recv[s] receives
source s’s rows. The GEMM reads A one K segment at a time: send[rank] for
this rank’s own columns, which never travel, and recv[s] for every other
source. One call with mode="fused":
Post. In block 0 of the GEMM, one lane per peer pushes
send[peer]inchunksrow bands, back to back, on the copy engine. Each band bumps a counter at the receiver once its bytes have landed.Compute. Tiles go band-major: the first tiles need only band 0 from each source, the next ones band 1, and so on. A tile starts on this rank’s own K segment, then takes the other sources in rank-rotated order.
Wait.
wait="segment": each wave waits for a segment’s band just before its first load of it, so the own segment overlaps the transfer.wait="tile": one thread waits for the tile’s band from every source before the tile starts. Block 0 takes the last tile, which needs the last band anyway.Close. A barrier-only kernel ends every call, so no rank refills a peer’s
recvwhile that peer still reads it. It also advances the epoch and re-takes the counter snapshot the band waits count from, both kept in the window: the next call, at any M and band count, and a CUDA-graph replay start from there.
mode="split" runs a push kernel, a barrier, the GEMM with no waits and the
closing barrier, over the same window. For the settings this means:
More bands let the first tiles start after less of the transfer, but each band has a fixed cost. On an idle GPU the README’s 70B exchange moves ~47 GB/s per link in one band and ~33 GB/s in eight.
With
wait="segment", what a tile can do before remote bytes arrive is its own segment: a quarter of K at P=4 and an eighth at P=8, too little at P=8 to cover the wait for the first bands.
3. Requirements
The shared contract (build and environment, window sizing, padding, result aliasing, the collective contract, cleanup, fp8 operands) is in What every operator has in common. For this operator:
Requirement |
Detail |
|---|---|
GPU |
gfx950 (MI355X). The GEMM is |
FlyDSL |
|
SDMA |
MORI built with |
Ranks |
P from 2 to 8, all in one node (one LSA domain). |
Communicator |
A |
4. Constructor
A2aGemmOp(comm, *, n, k, m_max, mode="fused", chunks=None, wait=None, acquire="inv")
Name |
Type |
Default |
Description |
|---|---|---|---|
|
|
required |
The communicator every rank shares: 2 to 8 ranks in one node, with SDMA queues. Its |
|
|
required |
N, the output columns and the rows of |
|
|
required |
K, the full reduction dimension (every head). A multiple of |
|
|
required |
The largest M (tokens per rank) the operator will see, at least 1. Rounded up to a multiple of 256 ( |
|
|
|
|
|
|
|
Bands per peer per call, 1 to 16. |
|
|
|
How a tile waits in |
|
|
|
What a tile does after a band has landed, in |
The constructor is collective: call it on every rank with the same arguments.
It checks its arguments before allocating, then allocates and zeroes the
window, creates a device communicator with one SDMA queue per peer, checks the
queues and the LSA domain, and snapshots the copy engines’ signal counters
between two host barriers. One instance serves any M up to m_max, and every
layer with this N and K: the weight is a call argument.
window_bytes_for()
A2aGemmOp.window_bytes_for(world_size, *, k, m_max) -> int is a static method.
Name |
Type |
Description |
|---|---|---|
|
|
P, the number of ranks |
|
|
As passed to the constructor; |
It returns the bytes the constructor allocates: 4 KiB of control plus
2 * M * K, with M the padded m_max, a send and a receive copy of the fp8
operand. The window does not depend on n, mode, chunks, wait or
acquire. The method does not validate its arguments. Size the communicator
with per_rank_vmm = 2 * window_bytes_for(...) + (256 << 20), as the overview
and the tests do.
5. Calling the operator
c = op(a_fp8, b_preshuffled, a_scale, b_scale) # torch.Tensor [M, N] bf16
Argument |
Shape |
Dtype |
Description |
|---|---|---|---|
|
|
|
Every token, this rank’s |
|
|
|
The replicated weight through |
|
|
|
One scale per token of this rank, over all of K. See below. |
|
|
|
One scale per output channel |
The A scale covers all of K. Each rank quantises only its own head
columns, but a token has one scale over every head, so every rank must
quantise token t with the same scale[t]. Compute it before quantising:
take each row’s absolute maximum over this rank’s heads ([P*M]), max-reduce
it across ranks, and derive the scale from that. Quantise
a[t] = attn_out[t] / scale[t] and pass this rank’s own tokens,
a_scale = scale[rank*M:(rank+1)*M]. The scale never travels. A scale
computed per head shard gives wrong results without an error.
Section 10 shows the computation.
Returns C [M, N] bf16 on the current GPU: this rank’s tokens,
[rank*M, (rank+1)*M) of the sequence. It is the operator’s buffer for that M
(one per M), and the next call with the same M writes it. Read it before this
rank’s next call, on the same stream or behind an event, or clone() it.
Ordering. The call is a collective: every rank makes the same calls, with
the same M, in the same order. The work goes on the current stream: a copy
into the window unless a_fp8 is send_buffer(M), the GEMM (which posts the
bands) and the closing barrier. "split" runs a push and a barrier first. The
send region may be refilled on the same stream as soon as a call is issued:
the closing barrier holds every rank until every GEMM has finished reading.
The first call at each M compiles its kernels. There are no call-time
options: mode, chunks, wait and acquire are fixed at construction.
Padding. Pad per destination. Rank d’s block, rows [d*M, (d+1)*M),
holds its real tokens first and zeros after them. Pad before computing the
scales. Zero rows come back as zero rows of C; slice c[:tokens].
CUDA graphs. A captured call replays like a fresh call. Warm each M
eagerly first, then refill the captured a_fp8 (or send_buffer(M)) and
a_scale in place before each replay, the same number of times on every rank.
6. Other methods
padded_m()
op.padded_m(m: int) -> int rounds a per-rank token count up to a multiple of
256: the M to pass. It raises ValueError for m < 1. It does not check
m_max; the call does.
send_buffer()
op.send_buffer(m: int) -> torch.Tensor returns the window’s send region for
M = m, [P*m, K/P] float8_e4m3fn. Rows [d*m, (d+1)*m) go to rank d.
Quantise the attention output straight into it and pass the same tensor as
a_fp8: the call then skips its copy into the window. Views for different m
share the same memory. Write into it on the stream of the previous call, or
from a stream that waits on that one. m must be a multiple of 256 and at most
m_max (ValueError).
self_test()
op.self_test() -> None is a collective check that the copy engines move
bytes. It fills every send slab with rank + 1, runs the split exchange at
m_max, and raises RuntimeError unless every slab received from a peer holds
that peer’s value: without SDMA, every put returns and moves nothing. Call it
once on every rank after construction, outside CUDA-graph capture (it
synchronises and runs host barriers), and before writing into send_buffer,
which it overwrites.
chunks_for() and wait_for()
op.chunks_for(m: int) -> int and op.wait_for(m: int) -> str return the
band count and wait mode a call at M uses. These are the constructor’s chunks
(rounded down to a divisor of M/256) and wait if given, otherwise
default_chunks(M, P) and default_wait(M, P). Log them to record what a
deployment runs with.
close() and the context manager
op.close() -> None releases the window and its memory. It is idempotent, and
leaving a with A2aGemmOp(...) as op: block calls it. It does not
synchronise: call torch.cuda.synchronize() first if a call may still be in
flight. The device communicator the operator created stays with the
communicator until that is destroyed. After close(), calling the operator,
send_buffer() or self_test() raises RuntimeError. A constructor that
fails releases the window before it raises.
Attributes
Attribute |
Meaning |
|---|---|
|
|
|
|
|
The window’s size, |
|
This rank and P, from the communicator |
|
The constructor’s arguments; |
7. Shapes and limits
Constraint |
Value |
Error if violated |
|---|---|---|
Ranks P |
2 to 8 |
|
One node |
|
|
SDMA queues |
at least 1 per peer |
|
|
multiple of 256 |
|
|
multiple of |
|
|
at least 1 |
|
Window span |
|
|
Per-rank VMM |
holds the window |
|
|
the values in section 4 |
|
|
|
|
|
2-D |
|
M per call |
multiple of 256, at most |
|
|
|
|
fp8 operands |
|
|
Scales |
|
|
Device |
every tensor on the GPU |
|
|
|
|
After |
— |
|
Not implemented: blockscale and mxfp8, N that is not a multiple of 256, a tile other than 256x256, and a per-shard A scale.
8. Choosing settings
default_chunks(m, world_size) and default_wait(m, world_size) pick the
settings per M. They are fitted on the README’s two shapes:
P |
Bands |
Wait |
|---|---|---|
2 to 7 |
|
|
8 |
1 up to M = 2048, then |
|
The band count is capped at 16 and rounded down to a divisor of M/256. At
the six measured points the defaults pick the best cell of the sweep in
section 9, except for the 70B shape at M = 4096. There they
pick tile c4 (259.0 µs) against the best, tile c2 (258.6 µs).
When to override:
Other shapes or rank counts. Only P = 4 and P = 8 were measured; P = 2, 3, 5, 6 and 7 use the P = 4 rule. Sweep
--chunksand--waitwith the benchmark (section 11) and pin the winners withchunks=andwait=.acquire="none"drops a cache invalidate after each wait. Every measurement here used it. It is correct while cco windows are uncached, the default; keep"inv"ifCCO_UNCACHED_WINDOW=0is set.mode="split"is the baseline and a fallback. It pushes inchunks_for(M)bands; the README’ssplit-sdmacolumn pushes one.
The operator reads no environment variables itself. These affect it:
Variable |
Effect |
|---|---|
|
Must be |
|
Unset (default): cco windows are uncached. |
|
Network interface for the communicator’s unique-id rendezvous; the tests and benchmarks use |
9. Performance
fp8 ptpc in, bf16 out, 256x256 tile. MI355X, one node, 2026-10-28, FlyDSL
0.4.0, torch 2.12.0+rocm7.14.0. Every case ran in one process on one
communicator and replayed its CUDA graph back to back, with an event between
launches, for at least 3 s. Each reports the median launch gap, maximum over
ranks; three rounds were interleaved (spread 0.0-2.0%). Every case was
validated against an fp32 reference at the fp8 floor (relL2 1.66e-3), with
acquire="none". gemm-only is the GEMM over an already exchanged operand;
split-rccl is all_to_all_single, then the same GEMM; split-sdma is push
(one band), barrier, the same GEMM, barrier. Times in µs:
Shape |
M per rank |
gemm-only |
split-rccl |
split-sdma |
best fused |
vs split-sdma |
vs RCCL |
|---|---|---|---|---|---|---|---|
MiniMax-M3 o-proj, P=4, N=K=6144 |
2048 |
63.0 |
144.0 |
132.9 |
121.7 |
-8.5% |
-15.5% |
4096 |
125.1 |
264.9 |
244.0 |
184.1 |
-24.5% |
-30.5% |
|
8192 |
208.6 |
479.0 |
438.7 |
300.2 |
-31.6% |
-37.3% |
|
Llama-3.1-70B o-proj, P=8, N=K=8192 |
2048 |
94.3 |
156.2 |
145.8 |
134.4 |
-7.8% |
-14.0% |
4096 |
182.6 |
294.6 |
269.6 |
258.6 |
-4.1% |
-12.2% |
|
8192 |
356.4 |
552.5 |
506.4 |
435.2 |
-14.0% |
-21.2% |
Every fused configuration of the same run (cN is N bands):
Shape |
M per rank |
wait |
c1 |
c2 |
c4 |
c8 |
|---|---|---|---|---|---|---|
M3 P=4 |
2048 |
segment |
121.7 |
128.2 |
136.8 |
157.5 |
2048 |
tile |
128.8 |
136.6 |
144.6 |
165.5 |
|
4096 |
segment |
234.0 |
184.1 |
190.8 |
209.4 |
|
4096 |
tile |
240.7 |
188.1 |
198.8 |
217.0 |
|
8192 |
segment |
424.0 |
356.1 |
300.2 |
317.6 |
|
8192 |
tile |
434.7 |
363.7 |
307.6 |
326.2 |
|
70B P=8 |
2048 |
segment |
134.4 |
139.0 |
149.0 |
176.0 |
2048 |
tile |
141.8 |
141.8 |
151.6 |
179.0 |
|
4096 |
segment |
262.4 |
259.7 |
261.4 |
263.0 |
|
4096 |
tile |
264.0 |
258.6 |
259.0 |
259.7 |
|
8192 |
segment |
515.2 |
499.8 |
466.0 |
448.3 |
|
8192 |
tile |
506.9 |
482.9 |
452.8 |
435.2 |
Through A2aGemmOp with its defaults (acquire="none", graph replay, input
written into send_buffer), the 70B shape at P=8 measured 136.6 / 262.0 /
430.9 µs fused and 149.7 / 290.1 / 557.4 µs split, at M = 2048 / 4096 / 8192
per rank.
Fusing wins in all six cells, by 4-32% against
split-sdmaand 12-37% against RCCL, most where the transfer is the larger half: at P=4 the exchange (split-sdmaminusgemm-only) is 70-230 µs against a 63-209 µs GEMM.The bands are the mechanism. With one band a remote segment waits for a peer’s whole slab; more bands let a wave start after a fraction of the transfer, and smaller bands lose bandwidth per put. At P=4 the best count rises with M: 1, 2, 4.
segmentis best or within 0.5% in five of six cells, and wins whenever the tiles fit in one wave (M = 2048).tileis ahead only at P=8 with M >= 4096, by 0.4-3%.At P=8 the fused path still trails the GEMM alone by 40-80 µs. The first wave of tiles needs half of the transferred bytes, and they arrive at the link’s rate. Closing that gap needs the bands posted before the GEMM starts, from attention’s epilogue, which is not done yet.
10. Complete example
# o_proj.py: MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=8 o_proj.py
import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.a2a_gemm import A2aGemmOp
from mori.ops.gemm_ar import preshuffle_b
N = K = 8192 # Llama-3.1-70B o-projection
M_MAX = 8192 # most tokens per rank this deployment will see
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo,cuda:nccl")
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
# Stand-in for the checkpoint's fp8 o_proj weight and per-channel scale.
g = torch.Generator(device="cuda").manual_seed(0) # the same on every rank
w_o = (torch.randn(N, K, generator=g, device="cuda") / 8).to(torch.float8_e4m3fn)
b_scale = torch.rand(N, generator=g, device="cuda") * 0.01 + 0.01 # [N] fp32
b = preshuffle_b(w_o)
tokens = 4000 # real tokens per rank
vmm = 2 * A2aGemmOp.window_bytes_for(world, k=K, m_max=M_MAX) + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with A2aGemmOp(comm, n=N, k=K, m_max=M_MAX) as op:
op.self_test()
m = op.padded_m(tokens) # 4096
a = op.send_buffer(m) # [P*m, K/P] fp8, in the window
a_scale = torch.empty(m, device="cuda") # [m] fp32
def prepare():
# Stand-in for Ulysses attention: every token, this rank's heads.
# Rank d's tokens are rows [d*m, d*m + tokens); the padding is zero.
attn = torch.randn(world, m, op.k_shard, device="cuda", dtype=torch.bfloat16)
attn[:, tokens:] = 0
attn = attn.view(world * m, op.k_shard)
amax = attn.abs().amax(dim=1).float() # [P*m], over this rank's heads
dist.all_reduce(amax, op=dist.ReduceOp.MAX) # over every rank's heads
fp8_max = torch.finfo(torch.float8_e4m3fn).max
scale = (amax / fp8_max).clamp_min(1e-12) # the same on every rank
quantize_rows_into(attn, scale, out=a) # your quantiser
a_scale.copy_(scale[rank * m:(rank + 1) * m])
prepare()
side = torch.cuda.Stream() # eager call on a side stream: it
side.wait_stream(torch.cuda.current_stream()) # compiles this M and warms it
with torch.cuda.stream(side):
c = op(a, b, a_scale, b_scale) # [m, N] bf16
y = c[:tokens].clone()
torch.cuda.current_stream().wait_stream(side)
# Capture, then refill and replay. thread_local: RCCL's watchdog thread
# may query events while this one captures.
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph, capture_error_mode="thread_local"):
c = op(a, b, a_scale, b_scale)
for _ in range(3):
prepare()
graph.replay()
y = c[:tokens].clone() # before the next call writes c
torch.cuda.synchronize() # nothing in flight at close()
dist.destroy_process_group()
quantize_rows_into stands for your quantiser; MORI ships none. It must write
a[t] = attn[t] / scale[t] as torch.float8_e4m3fn for all P*m rows, with
the scale every rank shares (unfused:
a.copy_((attn.float() / scale[:, None]).to(torch.float8_e4m3fn))). a_scale
is this rank’s m entries of that scale, fp32.
11. Testing and benchmarking
export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest -q tests/python/cco/test_a2a_gemm.py tests/python/cco/test_a2a_gemm_op.py
test_a2a_gemm.pychecks the window layout on the CPU. On 2 to 8 GPUs it runs the benchmark over every mode, 1 and 2 bands, both waits and bothacquirevalues with 8 back-to-back calls, then signal counters that wrap past 2^32 and band counts that change between calls.test_a2a_gemm_op.pyneeds 8 GPUs:self_test(),"fused"and"split"over several M on one instance (withsend_bufferand graph replays), back-to-back calls with one rank 20 ms late, and the constructor’s refusals, with SDMA on and off.
Allow about 10 s between back-to-back 8-rank runs so the SDMA queues are
released; test_a2a_gemm_op.py retries once when it hits this.
bench_a2a_gemm.py times gemm-only, split-rccl, split-sdma and
fused-sdma, driving the kernel builders directly rather than A2aGemmOp.
The MiniMax-M3 rows above came from this command; for the 70B rows, use 8 GPUs
with --nproc_per_node=8 -n 8192 -k 8192:
HIP_VISIBLE_DEVICES=0,1,2,3 MORI_SOCKET_IFNAME=lo MORI_ENABLE_SDMA=1 \
torchrun --standalone --nproc_per_node=4 \
benchmark/cco/flydsl/a2a_gemm/bench_a2a_gemm.py -m 2048 4096 8192 \
-n 6144 -k 6144 --chunks 1 2 4 8 --wait segment tile --acquire none \
--rounds 3 --out m3.jsonl
Flag |
Default |
Meaning |
|---|---|---|
|
all four |
Any of |
|
|
Tokens per rank after the exchange (several allowed), N and K |
|
|
Band counts and waits for |
|
|
|
|
|
Rounds over every case, alternating forward and reverse order |
|
|
Warm-up launches; the least number of launch gaps and seconds timed |
|
|
Adds that many calls with no host sync, one rank held before each, and validates every one |
|
|
The largest relL2 that passes; a JSONL file rank 0 appends one record per case to |
Each case also prints a RESULT_JSON line, and the script exits 1 if any case
fails validation. timeline_a2a_gemm.py, in the same directory, records when
each band lands and when each tile starts and ends; its docstring has the
command.
12. Troubleshooting
Error or symptom |
Cause |
What to do |
|---|---|---|
|
FlyDSL is not installed |
|
|
MORI built with |
Rebuild with |
|
The copy engines move no bytes |
As for the SDMA error above |
|
The ranks span nodes |
Run every rank on one node |
|
|
Size it from |
|
M above the window |
Construct with a larger |
|
|
Pass every token, with this rank’s |
|
The whole |
Pass |
A call never returns |
A rank skipped a call, used another M, or was built with another |
Same arguments and the same calls on every rank |
Wrong results, no error |
A per-shard A scale; the same scale slice on every rank; (fused) a sender with more |
|
The first call at a new M is slow |
It compiles that M’s kernels |
Warm every M before timing or capturing |
Further reading
The
a2a_gemmREADME, next to the code: the full design, every measurement, where the time goes at P=8, and the negative results.MORI-FUSED overview: what the six operators have in common, and the headline numbers.
GEMM + all-to-all, the QKV projection on the other side of Ulysses attention, and Running the tests and benchmarks.