Choosing an EP API and backend

MORI exposes the existing SHMEM-based EP API and the newer CCO-based EPv2 API. They have different configuration objects and initialization requirements. Keep an existing deployment on its tested API unless you are explicitly evaluating a migration.

EP paths in the development source

Path

Import / selection

Topology

Scope and restrictions

Existing EP

mori.ops

Intra-node and inter-node kernel types

SHMEM initialization; use MORI-EP User Guide for configuration, dispatch/combine, tuning and supported kernel types.

EPv2, FlyDSL

mori.ops.dispatch_combine_v2; kernel_backend="flydsl"

Intra-node only

Gather/scatter combine, quantization, standard-MoE conversion and routing replay, subject to dtype and architecture restrictions below.

EPv2, HIP

Same EPv2 import; kernel_backend="hip"

Intra-node

Gather combine in BF16/FP32. Dispatch can carry already-quantized FP8/FP4 payloads. Does not require FlyDSL.

EPv2, HIP internode preview

HIP backend with gpu_per_node < world_size

Multiple physical nodes over CCO/GDA

BF16/FP32/FP8 dispatch; no FP4, quantization, standard-MoE conversion or scatter combine. The communicator’s node grouping must match the config.

This table describes development source, not a guarantee that all paths are available in every published wheel. Check the installed release and its release notes.

EPv2 configuration and lifecycle

Import the EPv2 classes explicitly:

from mori.ops.dispatch_combine_v2 import (
    EpDispatchCombineConfig,
    EpDispatchCombineOp,
)

Select the backend with config.kernel_backend or MORI_V2_KERNEL_BACKEND. The FlyDSL backend is the default. Importing the package does not load either backend; construction loads the selected one. EPv2 uses an explicit CCO communicator, described in MORI CCO Guide. It does not inherit the legacy API’s process-global SHMEM bootstrap contract.

Read the maintained EPv2 package guide for configuration, initialization, teardown and complete commands. It documents the following important restrictions:

  • FP4 conversion kernels require gfx950 or gfx1250; gfx942 lacks the conversion intrinsics. The two supported targets use different instructions (gfx950 the cvt_scalef32_pk_*_fp4 pair-at-a-time family, gfx1250 a pack-of-8), so this is a per-target port rather than one shared path.

  • FP8 representation differs by target: OCP E4M3 on gfx950 and E4M3FNUZ on gfx942.

  • On the internode path, quant_type must be "none". The config rejects anything else whenever world_size > gpu_per_node, independently of the backend – the quantized-combine staging path there is incomplete. HIP is simply the only backend that has an internode path today.

  • gpu_per_node determines physical node grouping. A single host cannot emulate the two-node internode test by changing world size alone.

  • For internode, internode_kernel chooses "v2", "v2_ll" or "auto". Auto chooses by the current call’s token count, not buffer capacity.

Validation before tuning

Run the allocated-node checks in Quickstart before measuring performance. The source package also provides:

  • tests/python/ops/dispatch_combine_v2/test_ep_backend_parity.py for comparing HIP and FlyDSL on the same inputs.

  • tests/python/ops/dispatch_combine_v2/test_graph_capture.py for graph replay.

  • tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_internode.py for real two-node correctness, benchmarking, tuning and stress runs.

  • tests/python/ops/dispatch_combine_v2/bench_ep.py, whose benchmark points include an identity-expert correctness check by default.

Record the MORI/FlyDSL revisions, backend, GPU, NIC, transport, dtype, quantization mode, shape and graph mode alongside results. A successful intranode benchmark does not validate an internode configuration.