Communication APIs
MORI’s communication layers, each targeting a different use case.
MORI-EP (Expert Parallelism)
Dispatch and combine kernels for MoE expert parallelism.
Imports:
from mori.ops import (
EpDispatchCombineConfig,
EpDispatchCombineOp,
EpDispatchCombineKernelType,
)
Kernel Types:
Type |
Value |
Use Case |
|---|---|---|
|
0 |
Single-node EP via XGMI |
|
1 |
Multi-node baseline |
|
2 |
Multi-node optimized bandwidth |
|
3 |
Multi-node low-latency |
|
4 |
Async low-latency with pipelining |
|
5 |
Single-node low-latency, for small token counts |
EpDispatchCombineOp methods:
Method |
Description |
|---|---|
|
Route tokens to expert ranks. Returns (output, weights, scales, indices, recv_count). |
|
Combine expert outputs back. Returns (output, weights). |
|
Split dispatch for overlapping communication with computation. |
|
Split combine for overlapping communication with computation. |
|
Reset internal state between iterations. |
|
Get source positions of dispatched tokens (for verification). |
|
Get pre-registered zero-copy combine buffer. |
See MORI-EP Guide for full API reference.
AsyncLL v2 SDMA overlap
The HIP backend of mori.ops.dispatch_combine_v2 supports
async_ll=True, async_ll_transport="sdma". Run independent computation
between dispatch_send() and dispatch_recv(), and between
combine_send() and combine_recv(). Receive data and counts are ready
after the corresponding receive phase completes on the stream.
SDMA queues are shared per GPU pair by all ops in a process. When two ops
interleave on one stream (single-stream two-batch overlap), set
async_ll_own_sdma_queue=True on both so a receive does not wait for the
other op’s send. When the ops run on separate streams the option is not a
clear win: a two-op microbenchmark gets slower, while DeepSeek-V4 prefill with
two-stream two-batch overlap got faster. It is off by default; measure your
workload before turning it on there. Only the "sdma" transport uses it.
On a single gfx950 or gfx1250 node, the send phase now posts each payload followed by its ready notification on the same SDMA queue. The receive phase still waits for outgoing copies before the source buffers can be reused. Zero-token peers also exchange notifications. The redundant dispatch receive-count reset is omitted because the receive copy writes the total, including zero.
These options are read once when an op is constructed:
Environment variable |
Default and effect |
|---|---|
|
|
|
No longer has an effect. It skipped the host-side |
|
|
|
|
The first three variables accept 0 or 1. Skipping the reset or fusing
receive requires early notification. Set all three to 0 for the legacy sequence.
LSA and multi-node ops retain their existing protocol and ignore these SDMA
options. The resolved notification mode is part of every native Plan’s
configuration and generated source, so different ops can coexist without
changing the protocol of an already constructed Plan.
The default targets single-node gfx950 and gfx1250 overlap. On MI355X (gfx950, EP8, hidden 7168, top-6), early notification beat the legacy sequence at every token count from 128 to 4096: by 2-5% on one stream and 3-11% with two overlapping streams. Receive fusion remains opt-in because its additional benefit depends on the token count and compute window.
Receive fusion is a single-stream optimisation. A fused receive copy waits for its peers inside every CTA, so its whole grid stays resident for the wait. When two micro-batches overlap on two streams (two-batch overlap), that grid can crowd out the send pass the other stream must run before any peer’s wait can end, and the ranks deadlock. The grid ceiling makes this unlikely but cannot rule it out: it is sized from the compute units the process may use, not from those another stream, kernel or process is occupying at the time.
Do not combine receive fusion with micro-batches on two streams, especially when the device may be shared. On MI355X, two streams gained nothing from fusion over early notification, so dual-stream callers should use early notification alone, which is the gfx950 default.
MORI Shmem (Symmetric Memory)
OpenSHMEM-style APIs for GPU memory management and RDMA.
Imports:
import mori.shmem
Initialization:
Function |
Description |
|---|---|
|
Init from PyTorch process group (recommended) |
|
Init with broadcast unique ID |
|
Cleanup shmem resources |
Query:
Function |
Description |
|---|---|
|
Get current PE (rank) ID |
|
Get total number of PEs |
|
Get RDMA queue pairs per PE |
Memory Management:
Function |
Description |
|---|---|
|
Allocate symmetric GPU memory |
|
Aligned symmetric allocation |
|
Free symmetric memory |
|
Register existing buffer for RDMA |
|
Deregister buffer |
Communication:
Function |
Description |
|---|---|
|
Translate pointer to P2P address on remote PE |
|
Global barrier across all PEs |
See Shmem Guide for full API reference.
MORI-IO (Point-to-Point I/O)
RDMA-based P2P communication for KVCache transfer.
Imports:
from mori.io import (
IOEngine, IOEngineSession, IOEngineConfig,
BackendType, MemoryLocationType, StatusCode, PollCqMode,
RdmaBackendConfig, XgmiBackendConfig,
EngineDesc, MemoryDesc,
set_log_level,
)
IOEngine methods:
Method |
Description |
|---|---|
|
Get engine descriptor for remote registration |
|
Create an RDMA, XGMI, FABRIC or OFI backend (TCP only in simulation mode) |
|
Register a remote engine |
|
Register a PyTorch tensor for transfers |
|
One-sided read (remote → local) |
|
One-sided write (local → remote) |
|
Create reusable session for repeated transfers |
Enums:
BackendType:Unknown,XGMI,RDMA,TCP(simulation mode only),FABRIC,OFI(needs MORI built with libfabric)StatusCode:SUCCESS,INIT,IN_PROGRESS,ERR_INVALID_ARGS,ERR_NOT_FOUND,ERR_RDMA_OP,ERR_BAD_STATE,ERR_GPU_OP
See MORI-IO Guide for full API reference.
Device Bitcode Integration (mori.ir)
Framework-agnostic device bitcode and ABI metadata for integrating MORI shmem into GPU kernel frameworks (Triton, FlyDSL, MLIR, custom HIP, etc.).
Imports:
from mori.ir import (
find_bitcode,
get_bitcode_path,
MORI_DEVICE_FUNCTIONS,
SIGNAL_SET,
SIGNAL_ADD,
)
# Triton-specific (reference backend)
from mori.ir.triton import get_extern_libs, install_hook
Host-side API (framework-agnostic):
Function / Constant |
Description |
|---|---|
|
Locate |
|
Dict of all device function ABI metadata (symbol, args, ret types) |
|
Signal operation constants for put-with-signal (values 9, 10) |
Triton-specific API (reference backend):
Function |
Description |
|---|---|
|
Returns dict for Triton |
|
Install Triton compilation hook for automatic bitcode linking |
Key device functions (available as extern "C" symbols in bitcode):
Category |
Functions |
Description |
|---|---|---|
Query |
|
Get PE rank and total PE count |
P2P |
|
Translate pointers across PEs |
Put |
|
Non-blocking put at thread/warp/block scope |
Put + Signal |
|
Put with signal notification |
Atomics |
|
Atomic operations on remote memory |
Wait |
|
Spin-wait on remote memory values |
Sync |
|
Ordering and synchronization |
See the Shmem Guide for the full device function table and integration examples.