All-gather + GEMM
AgGemmOp (mori.ops.ag_gemm) computes C_r = concat_s(A_s) @ B_r.T: every
rank’s tokens times this rank’s weight shard, [P*M, N] in bf16. It serves a
ColumnParallelLinear after sequence parallelism, where each rank holds only its
own tokens. The copy engines push this rank’s rows to every peer in row bands.
The GEMM starts on its own rows, and each remote tile starts as soon as the
band it needs has landed.
Table of Contents
Quick Reference
from mori.cco import Communicator
from mori.ops.ag_gemm import AgGemmOp
from mori.ops.gemm_ar import preshuffle_b
# world, rank and uid (a mori.cco.UniqueId) come from your launcher: see section 10.
N, K, M_MAX = 6288, 7168, 4096 # per rank: weight-shard rows, hidden size, most tokens
vmm = 2 * AgGemmOp.window_bytes_for(world, k=K, m_max=M_MAX) + (256 << 20)
with Communicator.init(world, rank, uid, per_rank_vmm=vmm) as comm:
with AgGemmOp(comm, n=N, k=K, m_max=M_MAX) as op:
op.self_test() # collective, once
b_shuffled = preshuffle_b(b_fp8) # this rank's [N, K] shard, once
m = op.padded_m(x.shape[0]) # the same m on every rank
a_fp8, a_scale = quantize_per_token(op.pad_rows(x, m)) # your quantiser
c = op(a_fp8, b_shuffled, a_scale, b_scale) # [world*m, N] bf16
What |
Import |
|---|---|
Operator |
|
Weight layout |
|
Communicator |
|
Shape checks |
|
Limits (8 ranks, 16 bands) |
|
Kernel-level pieces the benchmark drives |
|
1. What it computes
Sequence parallelism leaves rank s with its own tokens A_s [M, K]. A
ColumnParallelLinear after it needs every rank’s tokens times this rank’s weight
shard B_r [N, K]. Unfused, that is an all-gather of the tokens, then the GEMM.
C_r = concat_s(A_s) @ B_r.T # [P*M, N]
C_r[s*M + i, j] = a_scale_s[i] * b_scale_r[j] * sum_k A_s[i, k] * B_r[j, k]
Per rank |
Shape |
Dtype |
Notes |
|---|---|---|---|
Tokens |
|
|
M padded to a multiple of 256, the same on every rank |
Token scale |
|
|
per token (ptpc); it travels with its rows |
Weight shard |
|
|
this rank’s output columns, through |
Weight scale |
|
|
per output channel |
Result |
|
|
contiguous; rows |
Every rank ends with all P*M tokens and its own N output columns, the layout a
column-parallel projection hands to the next tensor-parallel layer. The measured
case is Kimi-K3’s column-parallel projections at TP8: K=7168, and N per rank
6288 (KDA) and 3648 (MLA).
2. How it works
The window holds the gathered operand, so the GEMM reads one [P*M, K] tensor
whatever rank a row came from:
[ control | rows: [P][M][K] fp8 | scales: [P][M] fp32 ]
Slot r holds rank r’s rows. The call copies this rank’s rows into its own
slot, unless the caller quantised straight into it with local_rows(). In the
default mode="fused", a call is one GEMM kernel and one barrier kernel:
Block 0 pushes. One lane per peer pushes this rank’s rows and scales to that peer in
chunksrow bands, and sets an arrival flag in the peer’s window as each band lands. All peers are served at once; on MI355X each peer is its own xGMI link.Local tiles go first. Tiles over this rank’s own rows wait for nothing.
A remote tile waits for one band. The rest run in (band, source, row tile, column tile) order, with the source rotated by rank, so the first wave of remote tiles needs only each peer’s band 0.
A barrier closes every call. It advances the epoch the flags are compared against, so nothing is ever reset. No rank starts its next call until every rank’s GEMM has finished reading its slots.
mode="split" runs push, barrier, the same GEMM, barrier, and returns the same
bytes. More bands start the remote tiles earlier, but each band adds a wait
(section 8). The closing barrier is why every call is collective, why the next
call may use another M, and why a CUDA graph replays like a fresh launch.
3. Requirements
The operator follows the contract in What every operator has in common (build and environment, window sizing, padding, result aliasing, the collective contract, cleanup, fp8 operands). For this operator:
Requirement |
Detail |
|---|---|
GPU |
gfx950 (MI355X). The GEMM is |
FlyDSL |
|
Build |
MORI built with |
Environment |
|
SDMA queues |
|
Ranks |
P from 2 to 8, every rank in one node (one LSA domain). |
VMM |
|
Size the reservation from window_bytes_for() with the constructor’s k and
m_max; the examples reserve twice the window plus 256 MiB. With several
operators on one communicator, add up their windows: the operator test, which
keeps two instances alive at once, reserves
4 * window_bytes_for(...) + (256 << 20). The output buffer (section 5) is
ordinary PyTorch memory, outside the reservation.
4. Constructor
op = AgGemmOp(comm, *, n, k, m_max, quant="ptpc", mode="fused", chunks=None, sdma_queues=1)
Name |
Type |
Default |
Description |
|---|---|---|---|
|
|
required |
P is |
|
|
required |
Rows of this rank’s weight shard, which is its output columns. A positive multiple of 16. The last 256-column tile may be partial: past N it reads zeros and drops its stores, so a shard like 6288 goes in without padding the weight. |
|
|
required |
Hidden size. A multiple of 128, at least 256. |
|
|
required |
Largest per-rank M the instance will serve, at least 1. Rounded up to a multiple of 256 ( |
|
|
|
Only |
|
|
|
|
|
|
|
Row bands per peer in fused mode. |
|
|
|
SDMA queues to request per peer, at least 1. Peer |
All arguments after comm are keyword-only. The constructor is collective:
every rank calls it with the same arguments. It allocates and zeroes the window,
checks the SDMA queues and the LSA domain, and ends in comm.barrier(). If it
raises after allocating, it releases the window first.
window_bytes_for()
nbytes = AgGemmOp.window_bytes_for(world_size, *, k, m_max) # static
Returns the bytes the constructor will allocate, so the communicator can be
sized before the operator exists. world_size (int) is P, the communicator’s
nranks; k and m_max (int) are the constructor’s, and m_max is padded
to 256 the same way. The window is a 4 KiB control area, then P * m_max * K
bytes of fp8 rows and P * m_max * 4 bytes of scales, each rounded up to
4 KiB. n, mode, chunks and sdma_queues do not change it.
5. Calling the operator
c = op(a_fp8, b_preshuffled, a_scale, b_scale, out=None)
Argument |
Shape |
Dtype |
Notes |
|---|---|---|---|
|
|
|
This rank’s tokens. M is |
|
|
|
This rank’s weight shard through |
|
|
|
Per-token scale, M elements. Copied into the window unless it is |
|
|
|
Per-channel scale of this rank’s shard, N elements. |
|
|
|
Optional, contiguous. The result is written there and |
Every tensor must be on the GPU. The scales multiply back:
A ≈ a_fp8.float() * a_scale[:, None], and likewise each row of B with
b_scale.
Returns C_r, [P*M, N] bf16, rows [s*M, (s+1)*M) from rank s.
Without out= it is a view of the operator’s own output buffer, allocated once
at P * m_max * N bf16. That buffer is not the window, so the result survives
close(), but this rank’s next call overwrites it: consume it on the same
stream first (or behind an event the next call waits on), or clone it. out=
saves that clone.
The work goes on the current stream with no host synchronisation. Every rank
makes the same calls, with the same M, in the same order. The first call at
each M compiles its kernels. Rows that pad_rows() zero-filled come back as
zero rows, provided your quantiser maps a zero row to zeros with a finite
scale. With valid tokens on every rank, keep c.view(P, M, N)[:, :valid].
6. Other methods
padded_m()
op.padded_m(m) -> int returns m (at least 1) rounded up to a multiple of
256: the M to pass for m tokens per rank. It does not check m_max. Every
rank must pass the same M, so if token counts differ, pad every rank to the
padded M of the largest.
pad_rows()
op.pad_rows(x, m_pad) -> Tensor zero-extends a 2-D x [m, K] of any dtype to
m_pad >= m rows, in a buffer reused across calls, and returns a [m_pad, K]
view of it. The buffer grows when m_pad does, and the next pad_rows()
overwrites the view. Pad before quantising, so the scales cover the padded
rows. It copies even when x already has m_pad rows; skip it then.
local_rows()
op.local_rows(m) -> (rows, scales) returns this rank’s slot in the window at
m: rows [m, K] float8_e4m3fn and scales [m] fp32. Quantise straight into
them and pass them to the call, which then skips its copy into the window
(section 10 does this). Write into them on the stream you call the operator on,
after the previous call. They are valid until a call at another M, and not
after close(). self_test() overwrites the window, so run it first.
local_rows() at a new M sets that M up as a call would, and raises the same
errors.
chunks_for()
op.chunks_for(m) -> int returns the row bands per peer that fused mode uses
at a padded M, from the default rule or the pinned chunks (section 8). Split
mode always uses one. It is arithmetic only: use it to log what the operator
does, or to pass the same count to the benchmark’s --chunks.
self_test()
op.self_test() checks that the transport moves bytes and raises
RuntimeError if it does not. A MORI built with BUILD_CCO_SDMA=OFF, or run
without MORI_ENABLE_SDMA=1, still launches every kernel, but its puts move
nothing and the GEMM reads whatever the slots held. The test fills this rank’s
slot with rank + 1, runs the split push at op.m_max, and checks every
peer’s slot. It is collective: call it on every rank, once, after construction.
All ranks finish checking before any of them raises.
close() and with
op.close() releases the window, the compiled plans, and the output and
padding buffers. It is idempotent, and leaving a with AgGemmOp(...) as op:
block calls it. The device communicator is left to the communicator.
Synchronise first if a call may still be in flight, and do not replay a CUDA
graph that captured the operator once it is closed. Afterwards the call,
local_rows() and self_test() raise RuntimeError. The communicator holds a
reference to the window, so without close() it is freed only when the
communicator is destroyed.
Attributes
The constructor arguments stay readable as op.n, op.k, op.quant,
op.mode, op.chunks (None under the default rule) and op.sdma_queues.
op.m_max is the padded value, the largest M a call accepts. op.rank and
op.world_size come from the communicator, and op.window_bytes is what
window_bytes_for() returned.
supports() and shape_constraint()
supports(n, k, world_size) -> bool (from mori.ops.ag_gemm) is true when
2 <= world_size <= 8 and shape_constraint(n, k) (from mori.ops.ag_gemm.op)
returns None; otherwise shape_constraint returns the reason as a string.
They say whether a shape is expressible, not whether fused is faster.
7. Shapes and limits
Constraint |
Value |
Error if violated |
|---|---|---|
Constructor arguments |
|
|
K |
a multiple of 128, at least 256 |
|
N |
a positive multiple of 16 |
|
Ranks P |
2 to 8 ( |
|
Bands after rounding |
at most 16 ( |
|
One node |
every rank in one LSA domain |
|
SDMA queues |
at least |
|
Window |
fits in |
|
M per call |
a multiple of 256, at most |
|
M per call |
the same on every rank |
not checked: wrong results or a hang |
|
|
|
|
|
|
|
contiguous |
|
After |
|
The first use of an M is the first call, local_rows() or self_test() at
that M; self_test() uses op.m_max.
8. Choosing settings
Bands. With chunks=None, fused mode takes the most bands per peer, up to
4 (DEFAULT_CHUNKS), whose band still holds 20 output tiles (MIN_BAND_TILES):
the largest divisor d of the row tiles M/256 with d <= 4 and
(M/256/d) * ceil(N/256) >= 20, else 1. A thinner band costs more in waits
than its earlier start buys. The rule sees N only as its count of 256-column
tiles, so 6288 behaves as 6400, and it does not depend on P.
M per rank |
Row tiles |
N=6400 (25 column tiles) |
N=3840 (15 column tiles) |
|---|---|---|---|
256 |
1 |
1 |
1 |
512 |
2 |
2 |
1 |
1024 |
4 |
4 |
2 |
2048 |
8 |
4 |
4 |
4096 |
16 |
4 |
4 |
The rule was fitted on Kimi-K3’s TP8 projections at P=8, where it picks the
best measured count in every row of the tables in section 9. A pinned chunks
rounds down to a divisor of the row tiles: chunks=4 at M=768 (3 row tiles)
gives 3, and chunks=8 at M=1536 gives 6.
When to override.
Pin
chunksfor a shape far from the measured ones: sweep the benchmark’s--chunksat your M, N, K and P, and pin the winner. Above 4 is unmeasured.Use
mode="split"as the baseline to measure fused against, or as a fallback; it returns the same bytes.Leave
sdma_queues=1; the README has no measurement of other values.Set
m_maxto the largest per-rank M the deployment will see. It costs window (aboutP * m_max * Kbytes) and output buffer (P * m_max * Nbf16).
Environment variables. The operator reads none itself. MORI reads
MORI_ENABLE_SDMA once, when it sets up its context: without 1 the
communicator has no SDMA queues and the constructor refuses it.
MORI_SOCKET_IFNAME picks the interface for the communicator’s rendezvous;
the tests and the benchmark use lo on one node.
9. Performance
Kimi-K3’s column-parallel projections at TP8 (K=7168), fp8 ptpc in, bf16 out,
256x256 tile, on 8x MI355X (P=8) with ROCm 7.14, PyTorch 2.12 and FlyDSL
0.3.4.1. CUDA-graph replay, median of 51 after 20 warm-ups, maximum over ranks;
median of 2 interleaved rounds on an idle node (spread at most 1% except one
4.4% cell). All 92 runs validated at the fp8 floor (relL2 1.66e-3) against
every row of C for two operand sets. Global M is P times the M per rank. The
tables predate unpadded N, so they run the padded 6400 and 3840. split-rccl
is all_gather_into_tensor (rows, then scales) and the same GEMM; split-sdma
is push, barrier, GEMM, barrier; c1, c2, c4 are fused with 1, 2 and 4
bands per peer. µs:
N=6400 (KDA)
global M |
gemm-only |
split-rccl |
split-sdma |
fused c1 |
c2 |
c4 |
best vs split-sdma |
vs RCCL |
|---|---|---|---|---|---|---|---|---|
4096 |
169.8 |
257.5 |
258.7 |
260.6 |
237.9 |
- |
-8.0% |
1.08x |
8192 |
321.6 |
478.7 |
462.6 |
427.9 |
385.0 |
377.9 |
-18.3% |
1.27x |
16384 |
614.3 |
894.6 |
864.4 |
847.6 |
738.8 |
702.9 |
-18.7% |
1.27x |
32768 |
1195.8 |
1751.5 |
1693.1 |
1628.1 |
1397.4 |
1322.3 |
-21.9% |
1.32x |
N=3840 (MLA)
global M |
gemm-only |
split-rccl |
split-sdma |
fused c1 |
c2 |
c4 |
best vs split-sdma |
vs RCCL |
|---|---|---|---|---|---|---|---|---|
4096 |
104.0 |
188.2 |
178.8 |
176.8 |
190.4 |
- |
-1.1% |
1.06x |
8192 |
183.7 |
332.2 |
324.4 |
324.5 |
276.6 |
291.3 |
-14.8% |
1.20x |
16384 |
344.1 |
646.6 |
618.0 |
629.2 |
527.7 |
486.0 |
-21.4% |
1.33x |
32768 |
730.8 |
1316.6 |
1212.5 |
1182.1 |
985.5 |
896.9 |
-26.0% |
1.47x |
c4 needs four row tiles per rank, so it starts at global M=8192. The same fused c4 at Kimi-K3’s real shards and at the padded widths, 2 interleaved rounds, all validated, µs:
global M |
KDA 6288 |
padded 6400 |
MLA 3648 |
padded 3840 |
|---|---|---|---|---|
8192 |
374.8 |
373.3 |
271.6 |
283.3 |
32768 |
1300.3 |
1295.5 |
881.7 |
884.9 |
How to read them:
The bands are the mechanism. With one band every remote tile waits for a peer’s whole slab, so c1 stays close to split-sdma and is sometimes behind it. More bands start the first remote wave sooner; the best count rises with M.
The gain grows with M. The best fused call takes 7.6% (KDA) and 6.1% (MLA) less time than split RCCL at global M=4096, and 24.5% and 31.9% less at 32768. Part of the transfer stays exposed: 10.6-22.7% over the bare GEMM at 32768, and up to 70% at 4096.
Unpadded N is not faster, since the partial last tile still computes all 256 columns. It buys the interface: no padded copy of the weight, and a contiguous
[P*M, 6288]result instead of a slice of a 6400-wide one.
10. Complete example
# MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo torchrun --standalone --nproc_per_node=8 example.py
import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.ag_gemm import AgGemmOp
from mori.ops.gemm_ar import preshuffle_b
N, K = 6288, 7168 # this rank's shard rows (KDA at TP8) and the hidden size
M_MAX = 4096 # the most tokens one rank will pass
TOKENS = 1000 # tokens per rank in the step captured below
def main():
torch.cuda.set_device(int(os.environ.get("LOCAL_RANK", 0)))
dist.init_process_group(backend="cpu:gloo,cuda:nccl")
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
vmm = 2 * AgGemmOp.window_bytes_for(world, k=K, m_max=M_MAX) + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with AgGemmOp(comm, n=N, k=K, m_max=M_MAX) as op:
op.self_test() # every rank, once
w_fp8, w_scale = load_shard(rank) # [N, K] float8_e4m3fn, [N] fp32
b = preshuffle_b(w_fp8) # once per weight
def column_parallel(x): # x: this rank's [tokens, K] bf16
m = op.padded_m(x.shape[0]) # the same m on every rank
rows, scales = op.local_rows(m) # this rank's slot in the window
quantize_per_token_into(op.pad_rows(x, m), rows, scales) # your quantiser
return op(rows, b, scales, w_scale) # [world*m, N] bf16, no copy in
# Warm up outside the graph: compiles this M and allocates the
# operator's output and padding buffers. Every rank makes the same calls.
x = next_activations(0) # static [TOKENS, K] bf16 input
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
column_parallel(x)
torch.cuda.current_stream().wait_stream(side)
torch.cuda.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph, capture_error_mode="thread_local"):
c = column_parallel(x) # aliases the operator's buffer
m = op.padded_m(TOKENS)
for step in range(1, 4):
x.copy_(next_activations(step)) # refill the captured input
graph.replay() # every rank, in the same order
consume(c.view(world, m, N)[:, :TOKENS]) # same stream, before the next replay
torch.cuda.synchronize() # nothing in flight at close()
del graph # it addresses the window
dist.destroy_process_group()
if __name__ == "__main__":
main()
The placeholders are yours; MORI ships no quantiser. load_shard returns this
rank’s fp8 shard and its [N] fp32 per-channel scale. quantize_per_token_into
writes rows ([m, K] float8_e4m3fn) and scales ([m] fp32, one per token)
so that src[i] ≈ rows[i].float() * scales[i], turning a zero row into zeros
with a finite scale. next_activations and consume stand for the layers
around this one. thread_local capture follows the benchmark: it lets RCCL’s
watchdog thread query events during capture.
11. Testing and benchmarking
export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest -q tests/python/cco/test_ag_gemm.py tests/python/cco/test_ag_gemm_op.py
test_ag_gemm.py checks the window layout, then runs the benchmark in every
mode against an fp32 reference, back to back. test_ag_gemm_op.py exercises
the public operator in one torchrun launch, from self_test() and padding to
local_rows() with out= and calls with one rank held. The GPU tests need at
least 2 GPUs and run on up to 8.
The benchmark drives the kernels through compile_ag_gemm() and
build_phases(), not through AgGemmOp. Its --chunks defaults to 1 and is
not rounded, so it must divide m/256. The README’s command, for the global
M=32768 row of the KDA table (-m is per rank):
MORI_SOCKET_IFNAME=lo MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=8 \
benchmark/cco/flydsl/ag_gemm/bench_ag_gemm.py --mode fused-sdma --chunks 4 \
-m 4096 -N 6400 -k 7168 --back-to-back 8
Flag |
Default |
Meaning |
|---|---|---|
|
|
|
|
512, 6400, 7168 |
rows per rank, N, K |
|
1 |
bands per peer, for |
|
20, 51 |
timing loop; the median is reported |
|
off |
time eager launches instead of CUDA-graph replays |
|
0 |
R more calls with no host sync, a different rank held before each, each validated |
|
off, 3e-3 |
skip the fp32 reference, or set its relL2 threshold |
|
256, 2, 50000000 |
tile rows (only 256 is valid), the GEMM’s waves per EU, the held rank’s sleep |
Rank 0 prints one RESULT_JSON line (time as the maximum over ranks,
rel_l2, validated, remote_bytes_per_rank) and exits 1 if validation
fails. Before trusting a number of your own, read
Running the tests and benchmarks.
12. Troubleshooting
Error or symptom |
What it means |
|---|---|
|
Install the extra: |
|
MORI was built without |
|
The puts move nothing; check the same two settings. Without the test, calls would return wrong rows without an error. |
|
The ranks span nodes. The operator is single-node only. |
|
|
|
Only OCP e4m3fn is accepted; the launch would reinterpret other fp8 bytes. |
The GPU hangs in a call (the next synchronize never returns) |
A rank did not make the same call: a different number of calls, a skipped warm-up or replay, a different M, or a dead rank. Every call ends in a cross-rank barrier that spins on the GPU with no timeout. |
NaN or non-zero values in padding rows |
The padded rows were not zero after quantising. Pad with |
Results change after a later call |
The result is the operator’s buffer and the next call overwrites it. Clone it, pass |
The first call at a new M is slow |
Each M compiles once. Warm every M before timing or capturing. |
A launch right after another 8-rank run fails in |
The SDMA queues of a finished process group take seconds to come back. Wait about 10 s between back-to-back runs. |
Further reading
The
ag_gemmREADME, next to the code: the full design, every measurement, and the comparison with vllm-project/vllm#54151 (Kimi-K3 AG-GEMM on GB300).MORI-FUSED overview: the contract every operator shares and the headline numbers of all six.