GEMM + all-reduce
GemmAllReduceOp (in mori.ops.gemm_ar) multiplies each rank’s activations by
its shard of a row-parallel weight and all-reduces the product, so every rank
gets the [M, N] sum in bf16. It serves a RowParallelLinear under tensor
parallelism, such as DeepSeek-V4’s attention output projection wo_b. The GEMM
writes its C straight into the operator’s window, and the copy engine pushes
each destination’s slice as soon as its last tile is written, so the
reduce-scatter half of the all-reduce overlaps the rest of the GEMM. The reduce
and all-gather run after it. The same pipeline stopped before the gather is
GEMM + reduce-scatter.
Table of Contents
Quick Reference
from mori.ops.gemm_ar import GemmAllReduceOp, preshuffle_b
# comm: a mori.cco.Communicator whose per_rank_vmm was sized from
# GemmAllReduceOp.window_bytes_for(world, **settings); see section 10.
settings = dict(n=7168, k=2048, m_max=16384, quant="blockscale") # V4-Pro wo_b, TP8
with GemmAllReduceOp(comm, **settings) as op:
op.self_test() # collective, once per op
b_shuffled = preshuffle_b(b_fp8) # [N, K] e4m3fn, once per weight
m_pad = op.padded_m(x.shape[0])
a_fp8, a_scale = quantize_per_1x128(op.pad_rows(x, m_pad)) # your quantiser
out = op(a_fp8, b_shuffled, a_scale, b_scale)[: x.shape[0]].clone() # [M, N] bf16
torch.cuda.synchronize() # before close() frees the window
What |
Import |
|---|---|
The operator |
|
Weight and mxfp8 A-scale layouts |
|
Shape predicate |
|
Communicator |
|
Reduce-scatter variant |
|
GEMM and GEMV with nothing fused |
|
1. What it computes
Rank r of P passes its activations A_r and its weight shard B_r, and
every rank receives
out [M, N] bf16 = sum over r of dequant(A_r [M, K]) @ dequant(B_r [N, K])^T
K is the per-rank K, the slice of the layer’s input this rank holds; N is
not sharded. The GEMM applies the quant scales and accumulates in fp32, and
the reduce adds the P partials in fp32 and rounds once to bf16.
Shape |
dtype |
|
|---|---|---|
A ( |
|
|
B ( |
|
|
Scales |
per |
fp32, int32, uint8, or none |
Result |
|
bf16 |
With bf16 wires every rank gets the same bytes; with gather_dtype="fp8" the
bands a rank receives come through e4m3 and its own band does not. The layer is
a RowParallelLinear under tensor parallelism: the measured ones are the wo_b
of DeepSeek-V4-Pro (TP8) and DeepSeek-V4.1-Flash (TP4) and Kimi-K3’s MLP
down_proj; quant="bf16" serves layers a checkpoint keeps in BF16, such as
Kimi-K3’s attention o_proj.
2. How it works
A call issues four or more kernels on the current stream:
GEMM + scatter. The GEMM writes C into the window. Its rows split into
Pbands, one per destination, and each band into chunks; when a chunk’s last tile is written, the copy engine pushes the chunk into the destination’s receive slot. Tiles are walked chunk by chunk with the destination rotated by rank, so every link starts early.Drain: waits for this rank’s pushes to land, then a cross-rank barrier.
Reduce: sums the
Pcontributions to this rank’s band.Gather: sends the reduced band to every peer. An fp8 gather adds a quantise kernel, then a CU pull that widens on the way in (
"lsa") or a copy-engine push and a dequantise kernel ("sdma").
Only the scatter overlaps the GEMM, so the gain depends on how much GEMM there
is to hide it behind. A destination’s band is M / (P * block_m) tile rows,
which the default plan pushes in the largest number of chunks, up to 8, that
divides that count: more chunks start the first push earlier until the copy
engine’s per-packet cost shows, and 8 is the measured knee. At M = P * block_m
there is one chunk, pushed after the GEMM, and nothing overlaps.
The window holds C for m_max rows, the [m_max, N] bf16 result, a tmp slot
for this rank’s reduced slice (m_max / P rows), P receive slots of
m_max / P rows, an fp8 copy of the result when the gather is fp8,
and control words (barrier flags, a counter set per distinct M, submit locks),
all laid out for m_max. It is about 700 MiB per rank at m_max=16384,
N=7168, P=8 with bf16 wires, and 812 MiB with an fp8 gather.
3. Requirements
Requirement |
Detail |
|---|---|
GPU |
MI355X (gfx950). The GEMM kernels are written for gfx950 only and every measurement is on MI355X. The constructor does not check the GPU; the only architecture check in the code is |
FlyDSL |
|
SDMA |
MORI built with |
Ranks |
One node (one LSA domain), |
Memory |
A |
The rules this op shares with the other fused operators (padding, result aliasing, the collective contract, cleanup, fp8 operand and scale layouts) are in What every operator has in common.
4. Constructor
GemmAllReduceOp(comm, *, n, k, m_max, block_m=None, block_n=256, sdma_queues=1,
gather_dtype="bf16", gather_transport="lsa", quant="blockscale",
max_shapes=None, scatter_dtype="bf16", schedule="default")
Name |
Type |
Default |
Description |
|---|---|---|---|
|
|
(required) |
The communicator; |
|
|
(required) |
Output width N: a multiple of 256, and of 1024 with |
|
|
(required) |
Per-rank K. fp8 quants: a multiple of 128, at least 256. |
|
|
(required) |
Largest per-rank M the op will serve, at least 1. Rounded up to a multiple of |
|
|
|
Tile rows, which are also the padding granule. |
|
|
|
Tile columns. Only 256 is accepted: the epilogue’s lane transpose pairs exactly two N-tiles. |
|
|
|
SDMA queues per (source, destination) pair, at least 1. One is all the pipeline uses; more only adds idle queues, and in a process that already holds SDMA queues their creation fails. The first dev-comm on a communicator fixes the count. |
|
|
|
Wire dtype of the all-gather leg: |
|
|
|
Who moves an fp8 gather: |
|
|
|
Operand contract. |
|
|
|
Most distinct M plans the instance serves, at least 1; each takes a counter set. |
|
|
|
Wire dtype of the scatter leg: |
|
|
|
|
Construction is collective. It raises ValueError for a setting outside these
rules (section 7) before allocating anything, and
RuntimeError from the communicator if the window does not fit; on failure it
releases whatever it had acquired.
window_bytes_for()
GemmAllReduceOp.window_bytes_for(world_size, *, m_max, n, block_m=None,
gather_dtype="bf16", gather_transport="lsa", max_shapes=None, k=None,
quant="blockscale", scatter_dtype="bf16", schedule="default") -> int
Static. The bytes the window will take, by the constructor’s own planning rules:
op.window_bytes equals it for the same settings. world_size is
comm.nranks; the rest mean what they mean for the constructor, and k may be
omitted except with schedule="wo_b". block_n and sdma_queues do not change
the window. It raises ValueError for an unknown quant, a block_m the quant
forbids, a non-positive size, or a world size or fp8-gather N the window cannot
take; N and K against the GEMM are the constructor’s checks.
5. Calling the operator
out = op(a_fp8, b_preshuffled, a_scale=None, b_scale=None, *, logical_m=None)
Argument |
Shape |
dtype |
Notes |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
per |
per |
On the GPU; |
|
|
Keyword-only: the rows before padding. Selects the fitted plan with |
|
|
|
|---|---|---|
|
fp32, |
fp32 |
|
int32, flat: |
int32, flat: |
|
uint8 |
uint8 |
A blockscale a_scale that is [M, K/128] row-major is refused as ambiguous:
pass .reshape(-1) for a K/128-major buffer (what
aiter_per1x128_quant(transpose_scale=True) returns) or .t() for a logical
row-major tensor.
Returns an [M, N] bf16 row-major tensor that is a view of the op’s window,
holding the result once the work queued on the current stream has run. Every
M’s result starts at the same address, so this rank’s next call overwrites it
and close() frees it: order its readers before the next call (same stream, or
an event the next call’s stream waits on), and .clone() it to keep it. Padded
rows are zero; slice [:logical_m]. The first call at a new M compiles that M’s
kernels. One instance is not usable concurrently: the window, the kernel cache
and the padding buffer are shared.
6. Other methods
padded_m()
op.padded_m(m) -> int: the physical rows for m logical rows, m rounded up
to a multiple of P * block_m or, with schedule="wo_b", the fitted plan’s
rows at a profiled M (8200 gives 9216 at V4-Pro TP8, 4200 gives 4224 at
V4.1-Flash TP4). It keeps no state. ValueError if m < 1. The module
function padded_m(m, world_size, block_m=128) applies the default rule.
pad_rows()
op.pad_rows(x, m_pad) -> Tensor: zero-extends a 2-D [M, K] tensor to
m_pad rows in a buffer the op reuses and returns a view of it, which the next
pad_rows overwrites. Pad before quantising. The buffer is reallocated when a
call needs more rows or a different K, dtype or device, and a captured graph
keeps the old address, so run the largest M through it before capturing.
ValueError if x is not 2-D or m_pad < M.
self_test()
op.self_test(m=None), collective: writes rank + 1 over the input region,
runs scatter, reduce and gather without a GEMM, and checks every output element
is P * (P + 1) / 2 within 1%; otherwise it raises RuntimeError naming
BUILD_CCO_SDMA and MORI_ENABLE_SDMA. The logical m defaults to m_max,
which compiles what the first real call needs. Call it once on every rank after
construction and before any capture; it synchronises and overwrites the window.
close() and the context manager
op.close() deregisters the window and frees its memory. It is idempotent, and
a closed op raises RuntimeError. Dropping the object is not enough, because
the communicator holds the handles; the dev-comm and its SDMA queues stay with
the communicator. Synchronise first if work may be in flight.
with GemmAllReduceOp(...) as op: calls close() on exit.
Attributes
window_bytes (equal to window_bytes_for), m_max (padded) and the
constructor’s settings are plain attributes, for reading. dev_comm.sdma_num_queue
is the number of SDMA queues per peer the communicator actually connected.
7. Shapes and limits
supports(m, n, k, world_size, *, block_m=None, block_n=256, gather_dtype="bf16", gather_transport="lsa", quant="blockscale", scatter_dtype="bf16", schedule="default") takes a logical M and the constructor’s settings and
returns False where these rules fail, without constructing anything; it does
not apply the block_n == 256 rule or the bf16-scatter rule. It says whether a
shape is expressible, not whether fusing it is faster.
Constraint |
Value |
Error if violated |
|---|---|---|
|
|
|
|
256 for the mxfp8 quants; 128 or 256 for |
|
|
256 |
|
Wire and transport strings |
dtypes |
|
Scatter wire with |
|
|
|
|
|
|
at least 1 |
|
K, fp8 quants |
a multiple of 128, at least 256 |
|
K, |
a multiple of 64, at least 128 |
|
N |
a multiple of 256 |
|
N, |
a multiple of 1024 |
|
Ranks |
2, 4 or 8 (in [2, 8] and dividing 512) |
|
Call M |
|
|
Distinct M plans |
at most |
|
A and B |
exact shape; dtype per |
|
Scales |
dtype, element count and layout per |
|
Lifetime |
not closed |
|
Window |
fits the free VMM reservation |
|
8. Choosing settings
Setting |
What the default picks |
Change it when |
|---|---|---|
|
|
the checkpoint is mxfp8 or BF16. |
|
the quant’s tile: 128 for blockscale, 256 otherwise |
rarely: blockscale takes other granule-legal tiles, and bf16 also takes 128, which doubles the grid at small M; the mxfp8 forms need 256 |
Chunks |
the largest divisor of the bands per destination, up to 8 |
not a parameter; |
|
|
V4-Pro TP8 or V4.1-Flash TP4 |
|
|
the accuracy budget allows relL2 about 2.5e-2 for a gather half the size |
|
|
keep: the pull measured faster than the push on both targets |
|
|
V4.1-Flash TP4, once accuracy is checked: faster there, slower on V4-Pro TP8 |
|
1; every padded M up to |
keep |
The fp8 gather halves the bytes of the leg that runs at the xGMI ceiling. At V4-Pro TP8, M=16384, the fused layer measured 957.3 µs with the LSA pull, 1018.9 µs with the SDMA push and 1150.7 µs on the bf16 wire; relL2 against an fp32 reference went from 2.35e-3 to 2.49e-2. At the model output it was not detectable: V4-Pro’s perplexity on 10941 tokens was 3.2671, inside the 3.2547 to 3.2680 of three bf16 runs. No reasoning, code or maths benchmark has been run.
schedule="wo_b" applies only on gfx950, to two targets. At P=8, N=7168,
K=2048, "blockscale", block_m=128, both wires bf16: logical M=8200 keeps
9216 rows and pushes two bands at a time, and M=11264 and 13312 push one band
at a time. At P=4, N=5120, K=2048, "mxfp8", block_m=256: M=4200 and 8200
pad to a multiple of 64 (4224, 8256) and push one band at a time, with both
wires bf16, or with scatter_dtype="fp8" and a bf16 gather or an fp8 gather
over LSA; with both wires bf16, M=11264 also pushes two bands at a time. Every
other M, shape or setting keeps the default plan: it is a finite measured
profile, not an interpolation rule.
Whether to fuse at your M. At V4.1-Flash TP4 against the model’s unfused
path (section 9), the fp8 wire is ahead from M=2048 in every
row, and the bf16 wire is behind at 1024, 2048, 4200 and 4700 and ahead from
7200. SGLang’s integration fuses that layer from M=8192 on the bf16 wire and
from M=2048 on the fp8 wire, and V4-Pro’s from M=4096. Below that, run the GEMM
(or Mxfp8GemmOp) and your usual all-reduce.
Environment variable |
Default |
Effect |
|---|---|---|
|
unset |
Must be |
|
64 |
Grid of the LSA pull, so only with an fp8 gather over |
|
first non-loopback NIC |
Interface the communicator’s bootstrap uses; the tests set |
9. Performance
All from the README, on MI355X (gfx950), in µs, with the conditions given for each table.
wo_b at M=16384, maximum over ranks. V4-Pro is on 8x MI355X, median of 11.
V4.1-Flash is on 4x MI355X, from the README’s fp8-wire comparison; its fused
bf16-wire cell comes from a different run and is left out here:
DeepSeek-V4-Pro (TP8, blockscale) |
DeepSeek-V4.1-Flash (TP4, mxfp8) |
|
|---|---|---|
split SDMA, bf16 wire |
1472.9 |
1592.5 |
fused, bf16 wire |
1148.8 |
— |
fused, fp8 wire |
979.3 |
1192.1 (over LSA) |
GEMM + NCCL, as the model ships |
1419.5 |
— |
At smaller M on V4-Pro the order holds: 398.1 split, 351.0 fused and 329.5 fused with the fp8 gather at M=4096; 722.0, 621.1 and 539.3 at M=8192.
V4.1-Flash TP4 against what the model runs today,
mxfp8_native_blockscaled_linear and then an all-reduce (the route is
dot_scaled below M=8192 and hipblaslt_bf16 above). One M per process, two
runs; fill is M over the padded M; negative is fused faster:
M |
padded M |
fill |
today |
fused, bf16 wire |
fused, fp8 wire |
|---|---|---|---|---|---|
1024 |
1024 |
1.000 |
164.6 |
+16.4% |
+10.3% |
2048 |
2048 |
1.000 |
283.3 |
+1.8% |
-8.8% |
4096 |
4096 |
1.000 |
482.8 |
-0.9% |
-15.4% |
4200 |
5120 |
0.820 |
506.1 |
+17.0% |
-2.3% |
4700 |
5120 |
0.918 |
548.3 |
+8.4% |
-9.3% |
5120 |
5120 |
1.000 |
601.5 |
-5.8% |
-20.7% |
7200 |
8192 |
0.879 |
1051.2 |
-17.6% |
-32.7% |
8192 |
8192 |
1.000 |
1042.5 |
-19.4% |
-34.1% |
8200 |
9216 |
0.890 |
1185.1 |
-20.8% |
-35.7% |
9200 |
9216 |
0.998 |
1157.5 |
-19.2% |
-34.2% |
13000 |
13312 |
0.977 |
1547.8 |
-8.5% |
-25.3% |
16384 |
16384 |
1.000 |
1851.3 |
-16.2% |
-32.9% |
The opt-in settings: prequantised operands, 8 calls per graph, 21 replays, 3 alternating rounds (5 for the short tails). The 2026-09-29 runs used Torch 2.11, ROCm 7.2.4 and FlyDSL 0.2.4; the 2026-09-30 timings are from prototype modules since integrated into the op:
Setting |
Target |
Date |
Result |
|---|---|---|---|
|
V4-Pro TP8 |
2026-09-30 |
M=11264 1074.35 → 819.42 (-23.7%); M=13312 1274.86 → 957.80 (-24.9%); M=8200 728.82 → 702.48 |
|
V4.1-Flash TP4 |
2026-09-29 |
reduced padding: M=4200 533.56 → 460.27 (-13.7%); M=8200 894.46 → 830.37 (-7.2%) |
|
V4.1-Flash TP4, M=16384 |
2026-09-29 |
-21.1% with a bf16 gather, -26.6% with the fp8 gather over LSA; relL2 2.62% and 3.48% |
|
V4-Pro TP8, M=16384 |
2026-09-29 |
3.0% and 2.5% slower |
The two models do not reach the same conclusion. On V4-Pro the overlap dominates. On V4.1-Flash the GEMM is 178 µs against 1414 µs of communication, so most of the gain comes from the fp8 wire (-15.6% on its own) and the mxfp8 GEMM, not from fusing.
The fused cost follows the padded M while the baseline follows the true M, so one padded M can win at full fill and lose at low fill: read the whole curve. On Kimi-K3’s MLP
down_proj(blockscale, bf16 wires, TP8 and TP4, against the op’s own split path) fusing loses 2.0-4.0% at M=1024 and gains 9.7-21.6% at M=16384.In an SGLang server, V4.1-Flash prefill throughput at batch 4-16 rises +5.6% to +6.0% with the fused
wo_bon the fp8 wire and +7.4% to +7.7% with the standalone mxfp8 GEMM added. V4-Pro’s GPU-busy time over a profiled prefill falls 5.0%. Layer gains exceed these becausewo_bis about 12% of the profile.
10. Complete example
Run with MORI_ENABLE_SDMA=1 torchrun --standalone --nproc_per_node=8 example.py.
quantize_per_1x128 stands for your quantiser; MORI ships none. It must return
a_fp8 as [m_pad, K] float8_e4m3fn and a_scale as fp32 1x128 block scales:
[m_pad, K/128] column-major, [K/128, m_pad], or flat
(section 5).
import os
import torch
import torch.distributed as dist
from mori.cco import Communicator, UniqueId
from mori.ops.gemm_ar import GemmAllReduceOp, preshuffle_b
# DeepSeek-V4-Pro wo_b at TP8; add gather_dtype="fp8" for the shorter, lossy gather.
SETTINGS = dict(n=7168, k=2048, m_max=16384, quant="blockscale")
N, K = SETTINGS["n"], SETTINGS["k"]
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="cpu:gloo") # only carries the unique id
rank, world = dist.get_rank(), dist.get_world_size()
uid = [bytes(Communicator.get_unique_id()) if rank == 0 else None]
dist.broadcast_object_list(uid, src=0)
vmm = 2 * GemmAllReduceOp.window_bytes_for(world, **SETTINGS) + (256 << 20)
with Communicator.init(world, rank, UniqueId.from_bytes(uid[0]), per_rank_vmm=vmm) as comm:
with GemmAllReduceOp(comm, **SETTINGS) as op: # the same settings as the sizing
op.self_test() # every rank, once, before any capture
# This rank's weight shard; stand-ins for the checkpoint's tensors.
w_fp8 = (torch.randn(N, K, device="cuda") / 8).to(torch.float8_e4m3fn)
b_scale = torch.full((N // 128, K // 128), 1e-2, device="cuda")
b = preshuffle_b(w_fp8)
def wo_b(x): # x: [M, K] bf16, M <= m_max
m = x.shape[0]
a_fp8, a_scale = quantize_per_1x128(op.pad_rows(x, op.padded_m(m)))
return op(a_fp8, b, a_scale, b_scale)[:m]
# Warm every M you will run (the first call compiles it), largest first.
for m in (16384, 5000):
y = wo_b(torch.randn(m, K, device="cuda", dtype=torch.bfloat16)).clone()
# CUDA graph: a static input, warm-up on a side stream, then capture.
x_static = torch.randn(16384, K, device="cuda", dtype=torch.bfloat16)
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
for _ in range(3):
wo_b(x_static)
torch.cuda.current_stream().wait_stream(side)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
y_static = wo_b(x_static) # a view of the window
x_static.copy_(torch.randn_like(x_static)) # new data, same buffer
graph.replay() # on every rank, in the same order
result = y_static.clone()
torch.cuda.synchronize() # nothing in flight when close() frees the window
dist.destroy_process_group()
11. Testing and benchmarking
# Host-only, single-GPU, 2-rank (public op), 8-rank (kernels) and TP8/TP4
# (wo_b schedule) tests; cases needing more GPUs than the node has are skipped
BUILD_CCO_SDMA=ON MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
pytest tests/python/cco/test_gemm_ar.py tests/python/cco/test_flydsl_ar.py \
tests/python/cco/test_gemm_ar_op.py tests/python/cco/test_gemm_ar_schedule.py
# DeepSeek-V4-Pro, TP8
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
torchrun --standalone --nproc_per_node=8 benchmark/cco/flydsl/gemm_ar/bench_gemm_ar.py \
--mode fused-sdma --quant blockscale -m 16384 -n 7168 -k 2048
# DeepSeek-V4.1-Flash, TP4, with the wire that wins there
MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo \
torchrun --standalone --nproc_per_node=4 benchmark/cco/flydsl/gemm_ar/bench_gemm_ar.py \
--mode fused-sdma --quant mxfp8 -m 16384 -n 5120 -k 2048 \
--gather-dtype fp8 --gather-transport lsa --no-fuse-quantize
# Matrices, one point per process; the exit status counts failed points
python benchmark/cco/flydsl/gemm_ar/sweep.py --list
python benchmark/cco/flydsl/gemm_ar/sweep.py fused # writes fused.jsonl
python benchmark/cco/flydsl/gemm_ar/report.py fused.jsonl
tests/python/cco/test_gemm_ar_bench.py runs the benchmark as a test, and
gemm_ar_schedule_worker.py beside it runs the wo_b regression directly.
--mode takes gemm-only, split-rccl, split-sdma, fused-sdma,
split-lsa or fused-lsa; each run prints a RESULT_JSON line with
max_rank_time_us, rel_l2 and validated. Pin --quant (the default ptpc
is not an op setting), and note --sdma-queues 8 and --gather-transport sdma
against the op’s 1 and lsa. Allow about 10 s between back-to-back runs, and
read the README’s
Measurement traps
before trusting a small-M number of your own.
12. Troubleshooting
Symptom |
What it means |
|---|---|
|
The SDMA puts moved nothing: MORI was built with |
|
Install the extra: |
|
The window does not fit the communicator’s VMM reservation. Size |
|
A previous run’s ranks have not released their SDMA queues: wait about 10 s between runs (the tests wait 20 s) and check that no leftover process holds them. Keep |
|
M is not padded: use |
|
Say which you have: |
|
FNUZ has a different exponent bias; read as OCP e4m3 the product would be 4x too large. Quantise to |
|
An explicit |
A call hangs |
The phases carry device-side barriers: every rank must call with the same M, in the same order. Two unexplained hangs were seen at 8 ranks while a sweep was being set up; use |
|
Seen once, in one SGLang server out of nine, with the fused |
Standalone GEMM and GEMV
Mxfp8GemmOp is the same mxfp8 GEMM with nothing fused; Mxfp8GemvOp is a
separate skinny kernel for decode. Neither takes a communicator, a window or a
rank or needs SDMA (both need FlyDSL), and both return an ordinary tensor. They
are for a ColumnParallelLinear, which has no collective to fuse with, and for a
RowParallelLinear below the fusion’s useful M.
|
|
|
|---|---|---|
Call |
|
|
M |
|
1 to 32; above, |
Weight |
|
the same tensor |
Scales |
as the fused op’s |
uint8 |
Shapes |
K a multiple of 128, at least 256; N a multiple of 256; |
K a multiple of 128; N a multiple of 32 |
Predicate |
|
|
Kernels |
two: the 256-wide N tile, and a 128-wide one while the wide grid |
one per M bucket (1, 2, 4, 8, 16, 32); tuned for |
out=, if given, must be [M, N] bf16. On MI355X at V4.1-Flash wo_b
(N=5120, K=2048), M=16384, bf16 in and out with quantisation, the GEMM measured
196.0 µs against 281.3 µs for bf16 (fake_quant + hipBLASLt), -30.3%, and 274.9
µs for SGLang’s mxfp8 route (tl.dot_scaled); that was hot, one call per graph,
and bench_gemm.py --scope linear puts the comparison at -27%. Cold, across
V4.1-Flash’s fp8 linear shapes against SGLang’s native mxfp8 linear, what decides
it is the wide-tile grid ceil(M/256) * N/256: the median is +246.7% at 1-32
workgroups, -0.7% at 33-64, -10.2% at 65-128, -26.6% at 129-256 and -26.4%
above. On its two tuned shapes the GEMV beats SGLang’s mxfp8_gemv on the same
fp8 bytes at M=1, 8 and 32; six untuned shapes land within ±2%, wo_a is
12-16% slower at M=8 and above, and wq_b at TP8 is slower at M=32.
Further reading
The
gemm_arREADME, next to the code, has the full design, every measurement table and the negative results.MORI-FUSED overview: what the six operators share, and the headline numbers.