MORI Shmem Guide
MORI Shmem provides OpenSHMEM-style symmetric memory APIs for GPU memory management and RDMA communication. The original MORI-EP API (mori.ops.EpDispatchCombineOp) builds on it, so shmem must be initialized before that API is used; on gfx125x the original EP API defaults to CCO instead (MORI_EP_COMM). MORI-IO, MORI-CCO and the CCO-based EPv2 API do not use shmem.
Table of Contents
Quick Reference
import mori
# Initialize from PyTorch process group
mori.shmem.shmem_torch_process_group_init("default")
# Query
my_rank = mori.shmem.shmem_mype()
num_ranks = mori.shmem.shmem_npes()
# Allocate symmetric memory
ptr = mori.shmem.shmem_malloc(size_in_bytes)
# Register existing buffer for RDMA
mori.shmem.shmem_buffer_register(tensor.data_ptr(), tensor.nbytes)
# P2P address translation (intra-node)
remote_ptr = mori.shmem.shmem_ptr_p2p(ptr, my_pe, dest_pe)
# Synchronize
mori.shmem.shmem_barrier_all()
# Cleanup
mori.shmem.shmem_free(ptr)
mori.shmem.shmem_finalize()
Imports:
What |
Import |
|---|---|
All shmem APIs |
|
Init flags |
|
|
1. Concepts
Symmetric Memory
Symmetric memory is GPU memory that is allocated at the same virtual offset across all participating PEs (Processing Elements / ranks). This enables RDMA hardware to directly access remote GPU memory using simple address arithmetic — no address translation tables needed at runtime.
Processing Element (PE)
A PE is a participant in the symmetric memory domain. Each GPU rank maps to one PE. PEs are numbered 0 to N-1.
Symmetric Heap
The symmetric heap is a pre-allocated region of GPU memory from which shmem_malloc allocates. Its size is controlled by the MORI_SHMEM_HEAP_SIZE environment variable:
export MORI_SHMEM_HEAP_SIZE=6G # Must be set before shmem init
2. Initialization
Shmem must be initialized exactly once per process. There are three methods:
Method 1: PyTorch Process Group (Recommended)
Use this when PyTorch distributed is already initialized. The process group must be registered with a name before calling shmem_torch_process_group_init:
import torch
import torch.distributed as dist
import mori
# Standard PyTorch distributed init
torch.cuda.set_device(rank)
dist.init_process_group(
backend="cpu:gloo,cuda:nccl",
rank=rank, world_size=world_size,
device_id=torch.device("cuda", rank),
)
# Register the process group with a name
world_group = dist.group.WORLD
torch._C._distributed_c10d._register_process_group("default", world_group)
# Initialize shmem from the registered process group
mori.shmem.shmem_torch_process_group_init("default")
Note: The
_register_process_group("default", ...)step is required —shmem_torch_process_group_initlooks up the group by name to extract rank/world_size and bootstrap the shmem connections.
Method 2: Unique ID (No PyTorch Distributed)
Use this when PyTorch distributed is not available (e.g., standalone applications, custom launchers). Rank 0 generates a unique ID and broadcasts it to all ranks via any transport (file, MPI, TCP, etc.):
import mori
# Rank 0 generates a unique ID
if rank == 0:
unique_id = mori.shmem.shmem_get_unique_id() # Returns 128 bytes
# Broadcast unique_id to all ranks via your transport
# All ranks initialize with the same unique ID
mori.shmem.shmem_init_attr(
mori.shmem.MORI_SHMEM_INIT_WITH_UNIQUEID,
rank, # My rank
world_size, # Total ranks
unique_id, # Shared unique ID (bytes)
)
Example using file-based broadcast (for single-node testing):
import os, time
uid_file = "/tmp/mori_unique_id"
if rank == 0:
unique_id = mori.shmem.shmem_get_unique_id()
with open(uid_file, 'wb') as f:
f.write(unique_id)
else:
while not os.path.exists(uid_file):
time.sleep(0.1)
with open(uid_file, 'rb') as f:
unique_id = f.read()
mori.shmem.shmem_init_attr(
mori.shmem.MORI_SHMEM_INIT_WITH_UNIQUEID,
rank, world_size, unique_id,
)
Method 3: MPI Communicator (C++ only)
Use this in C++ MPI applications, with MORI built with MPI support
(MORI_WITH_MPI=ON). Pass the communicator through the attribute struct:
#include "mori/shmem/shmem_api.hpp"
mori::shmem::mori_shmem_init_attr_t attr{};
attr.mpi_comm = &comm; // an MPI_Comm
mori::shmem::ShmemInitAttr(mori::shmem::MORI_SHMEM_INIT_WITH_MPI_COMM, &attr);
Note: There is no Python path for this method:
mori.shmem.shmem_init_attrtakes a unique ID and cannot carry anMPI_Comm. In Python, use Method 1 or Method 2.
Initialization Comparison
Method |
When to use |
Dependencies |
|---|---|---|
PyTorch Process Group |
LLM inference/training frameworks (vLLM, SGLang, etc.) |
|
Unique ID |
Standalone apps, custom launchers, non-PyTorch environments |
None (any broadcast mechanism) |
MPI Communicator |
C++ MPI applications |
MPI, and MORI built with |
Finalization
Always finalize shmem before process exit:
mori.shmem.shmem_finalize()
# Then destroy PyTorch process group if applicable
dist.destroy_process_group()
3. Query APIs
# Get my PE (rank) ID — 0 to npes-1
my_pe = mori.shmem.shmem_mype()
# Get total number of PEs
total_pes = mori.shmem.shmem_npes()
# Get number of RDMA queue pairs per PE
num_qp = mori.shmem.shmem_num_qp_per_pe()
4. Memory Management
Allocating Symmetric Memory
# Basic allocation
ptr = mori.shmem.shmem_malloc(size_in_bytes)
# Aligned allocation (alignment must be power of 2)
ptr = mori.shmem.shmem_malloc_align(alignment=256, size=size_in_bytes)
# Allocation with flags
ptr = mori.shmem.shmem_ext_malloc_with_flags(size_in_bytes, flags)
All allocation functions return an integer address (int). The allocated memory is symmetric — the same offset is reserved on every PE.
Freeing Symmetric Memory
mori.shmem.shmem_free(ptr)
Registering Existing Buffers
Register a PyTorch tensor or other existing GPU memory for RDMA operations without allocating new symmetric memory:
# Register
tensor = torch.zeros(1024, 7168, dtype=torch.bfloat16, device="cuda")
mori.shmem.shmem_buffer_register(tensor.data_ptr(), tensor.nbytes)
# ... use tensor in MORI operations ...
# Deregister when done
mori.shmem.shmem_buffer_deregister(tensor.data_ptr(), tensor.nbytes)
Both functions return a status code (0 for success).
5. P2P Address Translation
For intra-node GPU-to-GPU access, translate a local symmetric pointer to its P2P-accessible address on a remote PE:
remote_ptr = mori.shmem.shmem_ptr_p2p(local_ptr, my_pe, dest_pe)
Return Value |
Meaning |
|---|---|
Non-zero |
P2P address on |
0 |
Connection uses RDMA transport (different nodes) or pointer is invalid |
6. Synchronization
Global barrier — blocks until all PEs reach the barrier:
mori.shmem.shmem_barrier_all()
7. HIP Module Init (Triton Integration)
When using Triton-compiled kernels that access shmem device symbols, initialize the HIP module:
mori.shmem.shmem_module_init(hip_module_handle)
This copies the current GPU states to the globalGpuStates symbol in the dynamically compiled Triton kernel module.
8. Initialization Flags
Flag |
Value |
Description |
|---|---|---|
|
0 |
Initialize using MPI communicator |
|
1 |
Initialize using broadcast unique ID |
9. Device Bitcode and IR
MORI-SHMEM device functions can be called from GPU kernels written in Triton, FlyDSL, MLIR or HIP. They are pre-compiled to LLVM bitcode (libmori_shmem_device.bc) as extern "C" wrappers. The mori.ir Python package locates the bitcode and gives the ABI metadata of every device function. Any framework that can link LLVM bitcode can use them. Triton is the reference integration. The package’s own guide, python/mori/ir/README.md, covers the integration layers and their tests.
For the device API of MORI-CCO (LSA, GDA, SDMA), see CCO IR.
Relationship with mori.shmem: The device functions in the bitcode depend on
the shmem runtime (globalGpuStates) being initialized on the host. Before any
device function can be called, the application must:
Initialize shmem via
mori.shmemhost APIs (shmem_torch_process_group_initorshmem_init_attr) — see 2. InitializationAllocate symmetric memory via
mori.shmem.shmem_mallocormori_shmem_create_tensorFor dynamically compiled kernels (e.g. Triton), call
mori.shmem.shmem_module_init(hip_module)to copy GPU states into the kernel module
mori.ir handles the compile/link-time concern (find bitcode, generate wrappers), while mori.shmem handles the runtime concern (init, memory, state).
Architecture
Compile / Link time Runtime (host)
────────────────── ───────────────
┌───────────────────────┐ ┌───────────────────────────┐
│ mori.ir │ │ mori.shmem (host APIs) │
│ find_bitcode() │ │ shmem_*_init() │
│ MORI_DEVICE_FUNCTIONS│ │ shmem_malloc/free │
│ mori.ir.triton (ref) │ │ shmem_module_init (JIT) │
└───────────┬───────────┘ └─────────────┬─────────────┘
│ locates & links │ initializes
┌───────────▼─────────────────────────────────▼─────────────┐
│ libmori_shmem_device.bc (LLVM bitcode) │
│ Device functions: put, atomics, wait, barrier, ... │
│ Depends on globalGpuStates (init by mori.shmem) │
└─────────────────────────┬─────────────────────────────────┘
│ called from
┌─────────────────────────▼─────────────────────────────────┐
│ Your GPU Kernel (Triton, FlyDSL, MLIR, HIP, ...) │
└───────────────────────────────────────────────────────────┘
Component |
Stage |
Role |
|---|---|---|
|
Compile/link |
Locate |
|
Compile/link |
ABI metadata dict for generating framework-specific wrappers |
|
Compile/link |
Reference backend: Triton-specific wrappers |
|
Runtime |
Initialize shmem, allocate memory, sync — required before device functions work |
Host-Side Python API
mori.ir (framework-agnostic)
These APIs have no dependency on any kernel framework — they work with Triton, FlyDSL, raw HIP, or any custom compiler:
Export |
Description |
|---|---|
|
Return absolute path to |
|
Alias for |
|
Dict of all device function ABI metadata — each entry contains |
|
Signal operation constant (value 9) |
|
Signal operation constant (value 10) |
Using ABI metadata to generate your own wrappers:
from mori.ir import MORI_DEVICE_FUNCTIONS
# Inspect a device function
func = MORI_DEVICE_FUNCTIONS["putmem_nbi_thread"]
print(func["symbol"]) # "mori_shmem_putmem_nbi_thread"
print(func["args"]) # ["uint64", "uint64", "uint64", "int32", "int32"]
print(func["ret"]) # "int32"
# Your framework can iterate MORI_DEVICE_FUNCTIONS to auto-generate
# wrappers for all 50+ device functions
mori.ir.triton (Triton-specific reference backend)
Export |
Description |
|---|---|
|
Returns dict for Triton |
|
Install Triton compilation hook for automatic bitcode linking |
Device wrappers |
All functions from |
Device Bitcode
libmori_shmem_device.bc is an LLVM bitcode library containing extern "C" device functions. To use it in your own framework:
from mori.ir import find_bitcode
bc_path = find_bitcode()
# bc_path is e.g. "~/.mori/jit/<arch>/<hash>/libmori_shmem_device.bc"
Then link it into your GPU kernel using your framework’s mechanism:
# Example: raw LLVM toolchain
llvm-link my_kernel.bc $(python -c "from mori.ir import find_bitcode; print(find_bitcode())") -o linked.bc
clang -target amdgcn-amd-amdhsa -mcpu=gfx942 linked.bc -o kernel.hsaco
Your device code can then call any function from the bitcode directly:
// In your HIP/device code
extern "C" int mori_shmem_my_pe();
extern "C" uint64_t mori_shmem_ptr_p2p(uint64_t dest_ptr, int my_pe, int dest_pe);
extern "C" int mori_shmem_putmem_nbi_thread(void* dest, const void* src, size_t n, int pe, int qp);
Device Functions
All functions below are available as extern "C" symbols in the bitcode. In Triton, they are callable as mori_shmem_device.<name>() inside @triton.jit kernels. In other frameworks, call the C symbol directly via bitcode linking.
Query
Function |
Args |
Return |
|---|---|---|
|
— |
|
|
— |
|
Point-to-Point
Function |
Args |
Return |
|---|---|---|
|
|
|
|
|
|
PutNbi (Thread / Warp / Block)
Function |
Args |
|---|---|
|
|
|
same |
|
same |
Typed variants: put_uint32_nbi_*, put_uint64_nbi_*, put_float_nbi_*, put_double_nbi_*
PutNbi with Signal
Function |
Args |
|---|---|
|
8 args |
|
same |
|
same |
Signal ops: SIGNAL_SET (9), SIGNAL_ADD (10)
Immediate Put
Function |
Args |
|---|---|
|
|
|
|
|
|
Atomics
Function |
Return |
|---|---|
|
|
|
|
|
|
|
|
Wait
Function |
Args |
Return |
|---|---|---|
|
|
|
|
|
|
|
|
|
Synchronization
Function |
Description |
|---|---|
|
Complete all pending remote operations (thread scope) |
|
Complete pending ops to specific PE |
|
Order remote operations (thread scope) |
|
Global barrier (thread scope) |
|
Global barrier (block scope) |
See python/mori/ir/ops.py for the complete function table with all C symbols.
Integration Example: Triton
Triton is the first reference backend. mori.ir.triton auto-generates Triton wrappers from MORI_DEVICE_FUNCTIONS:
import triton
import triton.language as tl
import mori.shmem as ms
from mori.ir import triton as mori_shmem_device
from mori.ir.triton import get_extern_libs, install_hook
# Host: initialize shmem
ms.shmem_torch_process_group_init("default")
buf = ms.mori_shmem_create_tensor((N,), torch.bfloat16)
install_hook()
# Device: Triton kernel
@triton.jit
def my_kernel(buf_ptr, N, BLOCK: tl.constexpr):
pe = mori_shmem_device.my_pe()
next_pe = (pe + 1) % mori_shmem_device.n_pes()
remote = mori_shmem_device.ptr_p2p(buf_ptr.to(tl.uint64, bitcast=True), pe, next_pe)
remote_ptr = remote.to(tl.pointer_type(tl.bfloat16), bitcast=True)
data = tl.load(remote_ptr + tl.arange(0, BLOCK))
my_kernel[(grid,)](buf, N, BLOCK=1024, extern_libs=get_extern_libs())
Integration Example: Raw Bitcode (no framework)
For custom compilers or HIP toolchains that don’t use Triton:
from mori.ir import find_bitcode, MORI_DEVICE_FUNCTIONS
bc_path = find_bitcode()
# Generate your own wrappers from ABI metadata
for name, meta in MORI_DEVICE_FUNCTIONS.items():
print(f"{meta['ret']} {meta['symbol']}({', '.join(meta['args'])})")
# Link bitcode into your kernel
llvm-link my_kernel.bc $(python -c "from mori.ir import find_bitcode; print(find_bitcode())") -o linked.bc
clang -target amdgcn-amd-amdhsa -mcpu=gfx942 linked.bc -o kernel.hsaco
Bitcode JIT Compilation
The bitcode is automatically JIT-compiled on first use — no manual build step required. find_bitcode() compiles shmem_device_api_wrapper.cpp with hipcc --cuda-device-only and caches the result to ~/.mori/jit/.
The NIC type (BNXT / AINIC / MLX5) and GPU architecture are auto-detected at runtime.
To precompile ahead of time:
MORI_PRECOMPILE=1 python -c "import mori"
Examples
File |
What it demonstrates |
|---|---|
|
Basic put/get via |
|
Allreduce: P2P read + put+signal kernels |
|
MLIR / LLVM IR paths (no Triton) |
Testing
# Triton basic tests (2 GPUs)
torchrun --nproc_per_node=2 examples/shmem/ir/test_triton_shmem.py
# Triton allreduce — P2P mode (8 GPUs)
torchrun --nproc_per_node=8 examples/shmem/ir/test_triton_allreduce.py
# Triton allreduce — IBGDA/RDMA mode (8 GPUs)
MORI_DISABLE_P2P=ON torchrun --nproc_per_node=8 examples/shmem/ir/test_triton_allreduce.py
# MLIR + LLVM IR paths (2 GPUs, no Triton)
cd examples/shmem/ir && bash run.sh 2 gfx942
Known Limitations
Triton’s
extern_elementwiseforces all device functions to returnint32even when the C function returnsvoid. This is a Triton upstream limitation.Pointer arguments are passed as
uint64(intptr cast) sinceextern_elementwisedoes not supportpointer_type(void).
10. Environment Variables
Variable |
Description |
Default |
|---|---|---|
|
Symmetric heap size (e.g., |
4 GB ( |
|
Heap mode: |
|
|
Heap memory type: |
|
|
VMM mode chunk size in bytes |
Auto |
|
Network interface for shmem bootstrap TCP connections (e.g., |
Auto-detect |
|
RDMA NIC selection. Include: |
All available |
|
mori loads libibverbs dynamically at runtime ( |
|
|
Number of RDMA queue pairs per PE |
|
|
InfiniBand GID index for RDMA connections |
Auto-detect |
|
RDMA service level |
Auto |
|
RDMA traffic class |
Auto |
|
mlx5 RoCE only. Resolve the QP’s destination MAC through an address handle, so that |
|
|
Disable P2P (XGMI) transport, force RDMA |
Not set |
|
Only create RDMA QPs to same-rail peers (same index within their node). For rail-isolated fabrics where cross-rail QPs cannot be established. See Rail-only connections. |
Not set |
|
Disable topology detection |
Not set |
|
By default the thread calling |
Not set (binding on) |
|
Bind to the whole NUMA node instead of this rank’s own slice of it. |
Not set (split on) |
|
Global log verbosity: |
|
|
Precompile all JIT kernels on import |
Not set |
|
Disable JIT compilation of device bitcode |
Not set |
11. Rail-only connections
By default Context builds a full mesh: every cross-node peer gets
MORI_NUM_QP_PER_PE queue pairs. On a rail-isolated fabric only NICs on
the same rail can reach each other, so the cross-rail QPs never reach RTR and
init hangs — even though no kernel would ever have used them.
MORI_ENABLE_RAIL_ONLY=1 restricts QP creation to same-rail peers. Two ranks are
on the same rail when they have the same index within their own node, which is
the same relation the EP internode kernels already encode in their proxy:
// src/ops/dispatch_combine/internode_v1.cpp
int proxyPe = i * config.gpuPerNode + (config.rank % config.gpuPerNode);
On a 2-node × 8-GPU job this takes each rank from 8 cross-node peers (32 QPs at
4 QP/PE) down to 1 (4 QPs). Skipped peers keep the same empty endpoint stubs
already used for non-RDMA peers, so rdmaEps indexing is unchanged; the
predicate is symmetric, so both ends stub each other and the handle AllToAll
stays aligned.
This is opt-in, and only safe for kernels whose cross-node traffic is same-rail.
Path |
Cross-node RDMA target |
Rail-only safe |
|---|---|---|
EP |
|
yes |
EP |
arbitrary |
no |
EP |
arbitrary |
no |
CCO with |
same rail |
yes |
CCO with |
arbitrary peer |
no |
Requirements checked at init — if any fails, mori logs an error and falls back to the full mesh rather than mis-pairing ranks:
node sizes are uniform,
ranks are node-major contiguous (
rankInNode == rank % gpuPerNode), which is what the EP proxy formula above assumes.
Same-host peers are never dropped: their RDMA loopback (used only under
MORI_DISABLE_P2P) does not traverse the fabric.
To confirm it took effect, look for this line from each rank:
rail-only: rank 3 rail 3 created QPs to 1 peers (4 QPs): 11
12. Source Files
File |
Description |
|---|---|
|
Python shmem API |
|
Public exports |
|
C++ shmem headers |
|
Python module entry point ( |
|
Shmem binding registration ( |