Running the tests and benchmarks
Every operator has a numerics test of its kernels and a multi-rank test of the
public operator (2 ranks for gemm_ar, 8 for the others). CI runs both in the CCO job. They need the environment
described in What every operator has in common:
export MORI_ENABLE_SDMA=1 MORI_SOCKET_IFNAME=lo
pytest tests/python/cco/test_gemm_ar.py tests/python/cco/test_gemm_ar_op.py
pytest tests/python/cco/test_gemm_a2a.py tests/python/cco/test_gemm_a2a_op.py
pytest tests/python/cco/test_gemm_ag.py tests/python/cco/test_gemm_ag_op.py
pytest tests/python/cco/test_ag_gemm.py tests/python/cco/test_ag_gemm_op.py
pytest tests/python/cco/test_a2a_gemm.py tests/python/cco/test_a2a_gemm_op.py
Each benchmark under benchmark/cco/flydsl/<op>/ checks its modes against a
reference and prints one RESULT_JSON line per configuration. All of them have
gemm-only, split-rccl, split-sdma and fused-sdma; gemm_ar adds
split-lsa and fused-lsa, gemm_a2a adds gemm-staging and gemm-to-window,
and gemm_ag adds gemm-to-window. Only bench_a2a_gemm.py runs all its modes
by default (--modes). The others run one --mode per invocation, by default
fused-sdma (gemm_ag: split-sdma). For example:
torchrun --standalone --nproc_per_node=8 \
benchmark/cco/flydsl/gemm_ar/bench_gemm_ar.py \
--mode fused-sdma --quant blockscale -m 16384 -n 7168 -k 2048
Each operator’s README has the exact commands behind its tables. Before trusting a number of your own:
Keep the node otherwise idle.
Interleave the configurations you compare across rounds.
Time CUDA-graph replays of many calls, not one.
Pin
--quant.Allow about 10 s between back-to-back 8-rank runs, so the SDMA queues are released.
The gemm_ar README’s
Measurement traps
lists the ones that have produced wrong numbers here.