GEMM + all-gather
GemmAgOp (mori.ops.gemm_ag) computes C = A @ B.T on each rank’s own rows
and gathers the results, so every rank gets every rank’s rows, [P*M, N]. It
serves a projection on a context- or sequence-parallel token shard whose output
every rank consumes, and was built for DeepSeek-V4’s compressor wkv_gate. The
GEMM writes C into this rank’s receive slot. On the fused path, which the default
picks at large M, its epilogue pushes each finished chunk of rows to every peer
over the SDMA copy engines while the GEMM computes the rest.
Table of Contents
Quick Reference
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ag import GemmAgOp
rank, world = dist.get_rank(), dist.get_world_size() # after torch.cuda.set_device()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
N, K, M_MAX = 2048, 7168, 16384 # DeepSeek-V4-Pro wkv_gate; M_MAX is rows per rank
vmm = 2 * GemmAgOp.window_bytes_for(world, n=N, m_max=M_MAX, out_dtype="fp32") + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with GemmAgOp(comm, n=N, k=K, m_max=M_MAX, in_dtype="bf16", out_dtype="fp32") as op:
op.self_test() # collective, once
m = op.padded_m(x.shape[0]) # x: this rank's [rows, K] bf16; the same m on every rank
c = op(op.pad_rows(x, m), w) # w: [N, K] bf16, the same on every rank
# c: [world*m, N] fp32; source s owns rows [s*m, (s+1)*m); padding rows are zero
What |
Import |
|---|---|
Operator |
|
Shape check |
|
Why a shape is rejected |
|
fp8 weight layout |
|
mxfp8 A-scale layout |
|
Communicator |
|
Window layout and kernel builders |
|
1. What it computes
Every rank r passes its own A_r [M, K] and the same B [N, K], and gets
back C = concat(A_0 @ B.T, ..., A_{P-1} @ B.T), [P*M, N], with rank r’s
rows at [r*M, (r+1)*M), its own included. P is the number of ranks and M
the padded rows per rank, the same on every rank. B must be the same on every
rank; the operator does not check it.
|
|
|
Scales |
Result (and wire) |
|---|---|---|---|---|
|
|
|
none |
|
|
|
|
by |
|
The layer is a projection on a context-parallel (CP) or sequence-parallel token
shard whose output every rank needs: in DeepSeek-V4’s wkv_gate each CP rank
scores its own tokens, then every rank needs every token’s score. The README’s
model shapes are DeepSeek-V4-Pro’s ratio-4 compressor.wkv_gate (K=7168,
N=2048, bf16 in, fp32 out) and DeepSeek-V4.1-Flash’s ratio-2 KV and gate
projections (K=5120, N=512 each, or 1024 combined); M is the runtime token count
per rank. SGLang’s Flash low-ratio CP path currently gathers the hidden states
before the projection, so for Flash this operator is a reordered candidate.
2. How it works
Each call launches two kernels on the current stream. The GEMM writes C straight into this rank’s slot of the receive buffer; every peer receives that same contiguous slab, so nothing is staged.
split: a one-block kernel then pushes the slab to every peer, one SDMA put each, waits for the copies and runs a cross-rank barrier.fused: the rows are cut into chunks of whole row tiles, one chunk per workgroup round of the GEMM grid, up tomax_chunks. When a chunk’s last tile is stored, the epilogue posts the chunk to every peer. Two chunks for one peer use different SDMA queues, so a fused call needs a queue per chunk per peer. The second kernel only waits for the queues and runs the barrier.auto(the default) picks one per M (section 8).
Fusion hides a chunk’s transfer behind the GEMM rounds still to come. At these shapes the transfer is several times the GEMM (Pro M=2048: 69 µs of GEMM against about 300 µs of all-gather), so fusion pays only at large M.
Receive-reuse credit. The result is the receive buffer, and the next call refills it. Each call first tells every peer it has started, and each push waits until its destination has started too, so a rank a call ahead never overwrites rows a slower rank is still reading. That holds provided whatever reads the result is ordered before this rank’s next call.
The window holds a few KiB of flags, chunk counters and credits, then the
receive buffer, P * m_max * N elements of out_dtype; it is allocated once
and cannot grow. A rank sends (P-1) * M * N * elem_bytes per call, 112 MiB at
Pro M=2048 with fp32 output and P=8. The wire carries the output dtype, so fp32
doubles the bytes of bf16.
3. Requirements
The contract every fused operator shares (build and environment, window sizing, padding, result aliasing, the collective contract, cleanup, fp8 layouts) is in What every operator has in common. For this operator:
GPU: MI355X (gfx950). The GEMMs are FlyDSL kernels written for gfx950 and every measurement is on MI355X; the operator does not check the architecture. It reads the current device’s CU count, so call
torch.cuda.set_device()before constructing it.FlyDSL:
pip install 'amd_mori[flydsl]'.import mori.ops.gemm_ag,preshuffle_band the layout helpers work without it;GemmAgOp,supportsand the kernel builders import it on first use.SDMA: a MORI built with
BUILD_CCO_SDMA=ON(the default), andMORI_ENABLE_SDMA=1in the environment when the communicator is created, which is when MORI reads it.SDMA queues:
mode="auto"and"fused"ask formax_chunksper peer (default 4),"split"for 1. The first device communicator on a communicator fixes its queue count, later ones reuse it, and MORI creates at most 8 per pair of GPUs. The constructor raisesRuntimeErrorif there are too few.Ranks: 2 to 8, all in one node (one LSA domain).
VMM: the window comes out of the communicator’s
per_rank_vmm; size it withwindow_bytes_for(section 4) plus every other window on the communicator. The examples reserve2 * window_bytes_for(...) + (256 << 20), as the overview does.
4. Constructor
GemmAgOp(comm, *, n, k, m_max, in_dtype="fp8", quant="ptpc", out_dtype="bf16",
mode="auto", max_chunks=4, chunks=None, tile=None, fence="release",
peer_uncached=True, war_credit=True)
Collective: every rank constructs with the same arguments. It validates them, allocates and zeroes the window, creates a device communicator with its SDMA queues, and ends with a barrier.
Name |
Type |
Default |
Description |
|---|---|---|---|
|
|
required |
Its ranks are the all-gather’s ranks. The only positional argument. |
|
|
required |
Output columns N, the weight’s rows. A positive multiple of 256. |
|
|
required |
Inner dimension K. fp8: a multiple of 128, at least 256. bf16: a multiple of 64, at least 128. |
|
|
required |
The most rows per rank any call will pass, at least 1. Rounded up to the M granule ( |
|
|
|
|
|
|
|
fp8 scaling: |
|
|
|
Result and wire dtype, |
|
|
|
|
|
|
|
The most chunks a fused call pushes in, and the SDMA queues per peer that |
|
|
|
Pins the fused chunk count instead of one per workgroup round; rounded down to a divisor of the row tiles, capped by |
|
|
|
Pins |
|
|
|
The fused epilogue’s fence before it pushes a chunk: |
|
|
|
Fused calls store C uncached ( |
|
|
|
The receive-reuse credit. |
chunks, fence and peer_uncached affect fused calls only.
window_bytes_for()
GemmAgOp.window_bytes_for(world_size, *, n, m_max, out_dtype="bf16", max_chunks=4) -> int
Static: the most bytes the constructor will allocate for its window, for sizing
the communicator’s VMM before the operator exists. world_size (2 to 8) is the
number of ranks; pass n, m_max, out_dtype ("fp32" doubles the receive
buffer) and max_chunks (it sizes the chunk counters) as the constructor gets
them. It rounds m_max up to 256, the largest M granule, so the value holds for
every in_dtype, quant, mode and tile. Raises ValueError for a world
size outside 2-8, an m_max below 1, or an n that is not a positive multiple
of 256.
5. Calling the operator
out = op(a, b, a_scale=None, b_scale=None) # -> torch.Tensor [P*M, N]
One GEMM + all-gather, issued on the current stream; it returns without
synchronising. M = a.shape[0] must be a positive multiple of the M granule
(use padded_m), at most op.m_max, and the same on every rank. The first
call at each M compiles that M’s kernels.
Argument |
|
|
|---|---|---|
|
|
|
|
|
|
|
|
required, by |
|
|
|
|---|---|---|
|
|
|
|
fp32 per 1x128 block: |
|
|
the flat |
the |
All operands must be GPU tensors; a and b are made contiguous (a copy if
they are not). A row-major [M, K/128] blockscale a_scale is rejected as
ambiguous: pass a_scale.reshape(-1) if its storage is already [K/128, M]
(as aiter_per1x128_quant(transpose_scale=True) returns it), or a_scale.t()
for a logical [M, K/128].
Return value: a contiguous [P*M, N] tensor in out_dtype, rank s’s
rows at [s*M, (s+1)*M). Zero-padded input rows come back as zero rows in
every source’s block; slice them off per source. It is a view of the
operator’s window, valid until this rank’s next call, self_test() or
close(). Order whatever reads it before that call (same stream, or an event
the next call’s stream waits on), or clone it.
6. Other methods
padded_m()
op.padded_m(m) -> int rounds a row count up to op.m_granule: 128, 256
under quant="mxfp8", or BLOCK_M when tile is pinned. Pad from a count
that is the same on every rank. ValueError if m < 1.
pad_rows()
op.pad_rows(x, m_pad) -> torch.Tensor zero-extends a 2-D [m, K] tensor to
[m_pad, K] in a buffer the operator reuses while m_pad fits and K, dtype
and device stay the same, so the next pad_rows overwrites it and a CUDA graph
that reads it sees each refill. Pad before quantising, so fp8 scales cover the
padded rows. ValueError if x is not 2-D or m_pad is too small.
fuses_at()
op.fuses_at(m) -> bool: whether a call at this M runs the fused path. Under
mode="auto" that is ceil(M/256) * ceil(N/256) >= op.cus; under "fused" or
"split" it is always True or False. No GPU work.
tile_for()
op.tile_for(m) -> tuple[int, int]: the (BLOCK_M, BLOCK_N) a call at this
padded M uses: the pinned tile, else for bf16 the per-shape rule in
section 8, else 128x256 (256x256 under mxfp8).
chunks_for()
op.chunks_for(m) -> int: the chunks a call at this padded M pushes in. 1 on
the split path; otherwise the pinned chunks or one per workgroup round of the
tile, capped at max_chunks and rounded down to a divisor of the row tiles.
self_test()
op.self_test() fills this rank’s slot with rank + 1, runs the split push at
m_max and checks every peer’s slot, raising RuntimeError at the first wrong
one; a put that silently moves nothing fails here. A collective: call it on
every rank once after construction, before capturing a graph. It synchronises
the device and overwrites the receive buffer.
close() and the context manager
op.close(), or leaving a with GemmAgOp(...) as op: block, releases the
window and drops the compiled plans and the pad_rows buffer. Idempotent.
Synchronise first if a call may still be in flight. Afterwards a call or
self_test() raises RuntimeError: this GemmAgOp has been closed.
Attributes
The constructor also sets, for reading: m_max (rounded up to the granule),
m_granule, window_bytes (this window’s size), sdma_queues (max_chunks,
or 1 under mode="split"), cus (the CU count the auto, tile and chunk
rules use, read at construction), rank and world_size.
supports()
supports(n, k, world_size, **kwargs) -> bool is True when
2 <= world_size <= 8 and shape_constraint(n, k, **kwargs) finds nothing;
kwargs are in_dtype, quant and out_dtype, with the constructor’s
defaults. It says whether a shape is expressible, not whether it is faster, and
looks at neither M, the GPU nor SDMA.
shape_constraint()
shape_constraint(n, k, *, in_dtype="fp8", quant="ptpc", out_dtype="bf16") -> str | None
returns why the constructor would reject this N, K and precision (it raises
ValueError("unsupported configuration: " + reason)), or None. Import it
from mori.ops.gemm_ag.op, which also exports the allowed values as
IN_DTYPES, QUANTS, OUT_DTYPES, MODES and FENCES.
preshuffle_b()
preshuffle_b(w) -> torch.Tensor reorders an fp8 [N, K] weight into the
order the fp8 GEMM reads; shape and dtype are unchanged. Call it once per
weight. It needs a 1-byte dtype, N a multiple of 16 and K a multiple of 64
(ValueError otherwise). bf16 weights are not shuffled.
Kernel-level exports
mori.ops.gemm_ag also exports what GemmAgOp is built from. AgConfig,
ag_config, counter_chunks and MAX_WORLD (8) are the window layout and
launch arithmetic, importable without FlyDSL. compile_bf16_gemm_ag,
compile_fused_gemm_ag, compile_gemm_local, build_sdma_phases and
build_sdma_chunk_post build the kernels. The benchmarks call them directly;
the operator’s users do not need them. build_sdma_chunk_post serves a
two-stream pipeline experiment and carries no receive-reuse credit.
7. Shapes and limits
Constraint |
Value |
Error if violated |
|---|---|---|
|
as in section 4 |
|
N |
a positive multiple of 256 |
|
K, fp8 |
a multiple of 128, at least 256 |
|
K, bf16 |
a multiple of 64, at least 128 |
|
|
the listed values |
|
|
at least 1 ( |
|
|
|
|
|
each 128 or 256, |
|
Ranks |
2 to 8 |
|
One node |
every rank in one LSA domain |
|
SDMA queues |
at least |
|
Open (each call) |
not closed |
|
M (each call) |
a positive multiple of |
|
|
|
|
Scales (each call) |
none for bf16; both for fp8 |
|
Scale dtype and size (each call) |
as in section 5 |
|
8. Choosing settings
The defaults were tuned on MI355X for the bf16-in, fp32-out wkv_gate shapes;
leave them unless a measurement of your own says otherwise.
mode="auto"fuses once even the largest tile’s grid covers the CUs,ceil(M/256) * ceil(N/256) >= CUs: a GEMM long enough for the transfer to hide behind. The rule fits every pair in the README’s results table at 256 CUs; a GPU with more CUs needs a larger M. Pin"split"when M stays small or SDMA queues are scarce (it asks for one per peer).bf16 tile, per M and per path, among 256x256, 128x256, 256x128 and 128x128. Split: the largest tile whose grid still covers the CUs, else the largest grid. Fused: the largest tile other than 256x256 whose grid runs at least two workgroup rounds, else the largest grid. fp8 uses 128x256 (256x256 under mxfp8).
Chunks: one per workgroup round of the fused tile, capped at
max_chunks. The cap defaults to 4 because 4 measured within 2% of 8 where the rule would pick 8, and a queue per chunk per peer is what runs out of SDMA engine slots first.fence,peer_uncached: the defaults are the README’ssc_release. On 2026-10-29 the benchmark’s default fence (leader) measured within 0.2% of it at M >= 4096, and 1.3% to 2.5% slower at M=2048. Do not use"none".war_credit: leave it on.No
max_shapes: one instance serves any number of distinct M up tom_max, in any order. Each new M compiles once; warm every M before capturing a CUDA graph.
What the defaults pick on MI355X (256 CUs), bf16 in, fp32 out, as fuses_at,
tile_for and chunks_for report it:
Shape |
M |
Path |
Tile |
Chunks |
|---|---|---|---|---|
Pro |
2048 |
split |
128x128 |
1 |
4096 |
split |
128x256 |
1 |
|
8192 |
fused |
128x256 |
2 |
|
16384 |
fused |
128x256 |
4 |
|
Flash combined, N=1024 |
2048 |
split |
128x128 |
1 |
16384 |
fused |
128x256 |
2 |
Environment variables. The operator reads none itself. MORI reads
MORI_ENABLE_SDMA when the communicator is created; MORI_SOCKET_IFNAME names
the network interface of the communicator’s bootstrap (the tests use lo).
9. Performance
From the
gemm_ag README,
on MI355X, in µs. Each table is its own session; compare within a table only.
Through the operator: Pro wkv_gate (N=2048, K=7168), bf16 in, fp32 out,
graph-timed, maximum over ranks. sc_release is the default publication;
leader is the leader fence.
P |
M |
|
|
|
|
|---|---|---|---|---|---|
8 |
2048 |
351.3 (split) |
362.8 |
361.0 |
351.3 |
8 |
4096 |
666.2 (split) |
685.5 |
687.6 |
666.1 |
8 |
8192 |
1239.2 (fused) |
1240.0 |
1240.9 |
1288.7 |
4 |
2048 |
349.1 (split) |
357.0 |
358.5 |
349.4 |
4 |
4096 |
662.0 (split) |
682.8 |
683.3 |
661.7 |
4 |
8192 |
1231.3 (fused) |
1231.3 |
1232.7 |
1282.2 |
Against Torch + RCCL: 2026-10-29, eight MI355X, ROCm
7.2, PyTorch 2.11.0+rocm7.2, FlyDSL 0.4.0; the benchmark’s modes, bf16 in, fp32
out, MORI’s GEMM at a fixed 128x128 tile; 50 warm-ups and 101 graph samples,
maximum over ranks, median of three interleaved rounds. Fused rows use
sc_release at the best chunk count (cN).
Shape |
M |
Torch + RCCL |
MORI GEMM + SDMA |
best fused SDMA |
best vs Torch + RCCL |
|---|---|---|---|---|---|
Pro |
2048 |
415.1 |
373.4 |
376.6 (c1) |
-10.0% |
4096 |
823.2 |
705.6 |
716.8 (c2) |
-14.3% |
|
8192 |
1510.5 |
1381.7 |
1314.8 (c4) |
-13.0% |
|
16384 |
3008.2 |
2844.2 |
2527.7 (c8) |
-16.0% |
|
Flash combined, N=1024 K=5120 |
2048 |
227.6 |
213.6 |
213.5 (c1) |
-6.2% |
16384 |
1465.9 |
1372.7 |
1309.2 (c4) |
-10.7% |
How to read them:
Fusion pays from M=8192 on Pro and at M=16384 on Flash N=1024, not below;
mode="auto"encodes that and takes the faster path in every row of the first table. At its own tile (128x256, a chunk per round) the fused path measured 3.9%, 7.4% and 2.2% under split SDMA at Pro M=8192, Pro M=16384 and Flash M=16384, on the box of the 2026-10-29 table.Against Torch + RCCL the steady part of the gain is the transport: SDMA’s all-gather alone is 12.5-13.9% faster than RCCL’s at every shape. The fixed 128x128 tile falls behind Torch’s GEMM at large M; the per-M tile closes that from M=8192.
The README’s only fp8 measurement, an earlier protocol at M=N=2048, K=7168, has one-chunk fusion behind split SDMA under ptpc, blockscale and mxfp8.
10. Complete example
bf16 in, fp32 out, the Pro wkv_gate shape, replayed from a CUDA graph:
# MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo torchrun --standalone --nproc_per_node=8 example.py
import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ag import GemmAgOp
N, K, M_MAX = 2048, 7168, 16384 # DeepSeek-V4-Pro wkv_gate; the most rows per rank ever passed
TOKENS = 3000 # rows per rank this step, the same on every rank
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo")
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
vmm = 2 * GemmAgOp.window_bytes_for(world, n=N, m_max=M_MAX, out_dtype="fp32") + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with GemmAgOp(comm, n=N, k=K, m_max=M_MAX, in_dtype="bf16", out_dtype="fp32") as op:
op.self_test() # collective, once
g = torch.Generator(device="cuda").manual_seed(0) # the same weight on every rank
w = (torch.randn(N, K, generator=g, device="cuda") / 8).to(torch.bfloat16)
torch.manual_seed(1 + rank) # a different shard per rank
x = (torch.randn(TOKENS, K, device="cuda") / 8).to(torch.bfloat16)
m = op.padded_m(TOKENS) # 3072
a = op.pad_rows(x, m) # the buffer the graph reads
print(rank, op.fuses_at(m), op.tile_for(m), op.chunks_for(m)) # MI355X: False (128, 128) 1
op(a, w) # warm-up: compiles this M
torch.cuda.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
out = op(a, w) # [world*m, N] fp32, in the window
for _ in range(3): # the same count on every rank
x = (torch.randn(TOKENS, K, device="cuda") / 8).to(torch.bfloat16)
op.pad_rows(x, m) # refills `a` in place
graph.replay()
result = out.view(world, m, N)[:, :TOKENS].reshape(world * TOKENS, N) # a copy
torch.cuda.synchronize() # nothing in flight at close()
dist.destroy_process_group()
fp8 in, bf16 out (the constructor’s default precision) with ptpc scales: as
above, but size the VMM for bf16 output (window_bytes_for(world, n=N, m_max=M_MAX)) and replace the operator block with
from mori.ops.gemm_ag import preshuffle_b
with GemmAgOp(comm, n=N, k=K, m_max=M_MAX, quant="ptpc") as op: # in_dtype="fp8" by default
op.self_test()
b = preshuffle_b(w_fp8) # [N, K] float8_e4m3fn, once per weight
m = op.padded_m(TOKENS)
a, a_scale = quantize_per_token(op.pad_rows(x, m)) # your quantiser
out = op(a, b, a_scale, w_scale) # [world*m, N] bf16
MORI ships no quantiser. quantize_per_token must return a as [m, K]
torch.float8_e4m3fn and a_scale as [m] fp32, one scale per row including
the padded rows; w_scale is the weight’s [N] fp32 scale. For blockscale
and mxfp8, produce the layouts in section 5.
11. Testing and benchmarking
Run against an installed MORI: the checkout’s python/ has no compiled
mori.cco.cco, so keep it off PYTHONPATH.
export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest tests/python/cco/test_gemm_ag.py tests/python/cco/test_gemm_ag_op.py
test_gemm_ag.py covers the layout arithmetic, the tile and chunk rules,
models of the LDS ordering and SDMA completion, and single-GPU GEMM numerics
(skipped without gfx950). test_gemm_ag_op.py checks the configuration rules,
then runs one 8-rank torchrun of gemm_ag_op_worker.py: every precision in
both modes, every bf16 tile, chunk counts, back-to-back calls with a held
reader with the credit on and off, alternating M, padding, operand checks,
self_test and close. The multi-rank parts need 8 GPUs and FlyDSL.
The benchmark benchmark/cco/flydsl/gemm_ag/bench_gemm_ag.py calls the
kernels, not GemmAgOp, and defaults to the leader fence with cached stores.
With --block-m, --block-n and --chunks left at 0 it picks the tile and
chunks by the operator’s rules at 256 CUs, with up to 8 chunks. Fused SDMA at
the Pro shape, as the operator runs it:
python -m torch.distributed.run --standalone --nproc_per_node=8 \
benchmark/cco/flydsl/gemm_ag/bench_gemm_ag.py \
--mode fused-sdma --peer-uncached --fence release \
--in-dtype bf16 --out-dtype fp32 -m 8192 --out-dim 2048 -k 7168 \
--rounds 5 --warmup 50 --iters 101 --tolerance 1e-5
--mode also takes gemm-only, gemm-to-window, split-rccl and
split-sdma (--fence release is refused outside fused-sdma). --in-dtype
defaults to fp8, with --quant ptpc|blockscale|mxfp8 and a default
--tolerance of 3e-3. --torch-gemm swaps in torch.mm for bf16-in,
fp32-out gemm-only, split-rccl and split-sdma; --phase-split also times
the GEMM alone. Each run prints RESULT_JSON and TIMING_JSON; the time is
the median over rounds of the per-round maximum over ranks. Run one
configuration at a time on an idle node, and let one process group’s SDMA
queues tear down before the next. The
benchmark README
also covers the two-stream pipeline experiment and the GEMM-only comparison.
12. Troubleshooting
Error or symptom |
What it means |
|---|---|
|
No SDMA: MORI was built without |
The same error with a non-zero count |
An earlier device communicator on this communicator fixed a smaller queue count. Construct the operator that needs the most queues first, lower |
|
The ranks span nodes; the operator is single-node. |
|
The puts complete but move nothing. Check |
|
Usually the window does not fit in what is left of |
|
Pad with |
|
Construct with a larger |
|
Wrong operand dtype, for example |
|
Pass |
Wrong numbers, no error |
An fp8 |
The result changes after the call |
It was read after this rank’s next call (or |
A hang |
The ranks disagree on the collective contract: a different number of calls or graph replays, a different M, or only some ranks constructing or calling |
|
The SDMA queues of the previous multi-rank run are not released yet. Wait and retry; the tests wait for the previous run’s processes to leave the GPU, then retry once. |
Further reading
The
gemm_agREADME, next to the code: the full design, every measurement, and what was tried and dropped (LSA transports, a short final chunk, smaller tiles, a lossy wire, a two-stream pipeline).MORI-FUSED: Fused GEMM + Collective Operators: what the six operators share, and performance at a glance.
Running the tests and benchmarks for every fused operator.