MORI UMBP PD Disaggregation Benchmark

Runs a prefill-decode disaggregated serving benchmark using SGLang with mori’s UMBP KV-cache transfer backend.

This guide is xP1D-generic: N prefill nodes (N >= 1) plus one decode node. The 1P1D case is just N=1; the 2P1D case is N=2. Set PREFILL_NODES / PREFILL_IPS to drive the topology — every step below loops over them.

For the architecture this benchmark exercises (master-as-advisor, heartbeat-event index, peer-owned allocator), see src/umbp/doc/design-master-control-plane.md. Every UMBP_* env var referenced below is documented in src/umbp/doc/runtime-env-vars.md.

Table of Contents

Prerequisites

  • N+1 nodes with ROCm-capable GPUs (8 GPUs per node, dp8ep8 topology): N prefill nodes + 1 decode node (N >= 1)

  • NFS mount shared between all nodes (for logs, results, and mori/sglang source)

  • SSH key access from the local machine to all nodes

  • Docker with ROCm image available on all nodes

  • mori built with UMBP=ON (Step 3 below)

  • SGLang checked out at $NFS_BASE/sglang with benchmark/hicache/run_pd_disagg_bench_dp8ep8.sh supporting both PREFILL_URLS (space-separated N prefill URLs) and the wait_for_router_workers gate (polls /get_loads after router is ready)

Topology

┌──────────────────────────────────┐
│       PREFILL NODE [0] (PRIMARY) │
│                                  │
│  SGLang prefill server :30000    │
│  UMBP master           :15558  ◄────────────┐
│  KV events publisher   :5557     │          │
└──────────────────────────────────┘          │
                                              │
┌──────────────────────────────────┐          │   ┌──────────────────────────────────┐
│       PREFILL NODE [i] (i>=1)    │          │   │          DECODE NODE             │
│  (only present when N >= 2)      │          │   │                                  │
│                                  │          │   │  SGLang decode server  :30001    │
│  SGLang prefill server :30000    │          ├──►│  sglang_router         :8000     │
│  UMBP_MASTER_AUTO_START=false  ──┼──────────┤   │  benchmark client            ────┼─► localhost:8000
│  KV events publisher   :5557     │          │   │  UMBP_MASTER_AUTO_START=false ───┘
└──────────────────────────────────┘          │   │  KV events publisher   :5557     │
                                              │   │                                  │
                                              │   │  Grafana       :3000             │
                                              │   │  Prometheus    :9090             │
                                              │   └──────────────────────────────────┘
                                              │
                                  (UMBP master discovery / dp-rank registry)

All nodes share a single UMBP master that runs on PREFILL_NODES[0] (the primary prefill). The primary’s bench script auto-starts the master; all other prefills (i>=1) and the decode node set UMBP_MASTER_AUTO_START=false and connect to it. With N prefills + 1 decode this means (N+1) * 8 = 8(N+1) dp ranks all register with one master.

The decode node hosts sglang_router (started by the bench script). The router fans out to all N prefills via repeated --prefill flags driven by the PREFILL_URLS env var.

Configuration

Define these variables once before running any step. All subsequent code blocks refer to them. PREFILL_NODES and PREFILL_IPS are bash arrays of equal length — set one entry for 1P1D, two for 2P1D, N for xP1D.

# === Edit for your environment ===
USER_HOME="/home/youruser"             # home directory (same path on all nodes via NFS)
NFS_BASE="/nfs/users/youruser"         # NFS root containing sglang/ and mori/
SSH_KEY="$USER_HOME/.ssh/id_ed25519"   # SSH private key for node access

# Prefill nodes (xP1D). Index 0 is the PRIMARY: it hosts the UMBP master and
# must be started first. Indices 1..N-1 are SECONDARY: they connect to the
# primary's master. Both arrays MUST have the same length.
#
#   1P1D example:
#     PREFILL_NODES=("node-prefill-1")
#     PREFILL_IPS=("10.x.x.1")
#
#   2P1D example:
#     PREFILL_NODES=("node-prefill-1" "node-prefill-2")
#     PREFILL_IPS=("10.x.x.1" "10.x.x.2")
PREFILL_NODES=("node-prefill-1")
PREFILL_IPS=("10.x.x.1")

NODE_DECODE="node-hostname-decode"     # decode node hostname
IP_DECODE="10.x.x.y"                   # decode node IP

# Set to the image matching your AMD GPU platform, e.g.:
#   MI300X / MI325X (gfx942): rocm/sgl-dev:vX.Y.Z-rocm7XX-mi30x-YYYYMMDD
#   MI350X        (gfx950):   rocm/sgl-dev:vX.Y.Z-rocm7XX-mi35x-YYYYMMDD
DOCKER_IMAGE="rocm/sgl-dev:v0.5.9-rocm700-mi30x-20260316"

# Network interfaces — find with: ibstat | grep CA; ip link show
NCCL_IB_HCA="ionic_0,ionic_1,ionic_2,ionic_3,ionic_4,ionic_5,ionic_6,ionic_7"
NET_IFNAME="ens14np0"          # TCP socket interface for GLOO/NCCL/MORI
MORI_RDMA_DEVICES="^mlx5_8"   # use ^ prefix to exclude; leave empty to use all devices

# Benchmark parameters
OUTPUT_LENGTH=100              # decode output tokens per request

# Extra Docker volume mounts (site-specific; clear if not needed)
EXTRA_MOUNTS=""
# Example: EXTRA_MOUNTS="-v /data/models:/models -v /apps:/apps"
# =================================

# === Derived (do not edit) ===
# Primary prefill: hosts the UMBP master.
NODE_PREFILL_PRIMARY="${PREFILL_NODES[0]}"
IP_PREFILL_PRIMARY="${PREFILL_IPS[0]}"
N_PREFILLS=${#PREFILL_NODES[@]}

# Sanity check arrays are same length and non-empty.
if (( N_PREFILLS == 0 )) || (( N_PREFILLS != ${#PREFILL_IPS[@]} )); then
    echo "ERROR: PREFILL_NODES and PREFILL_IPS must be non-empty and same length" >&2
    return 1 2>/dev/null || exit 1
fi

SSH="ssh -o StrictHostKeyChecking=no -i $SSH_KEY"
SCP="scp -o StrictHostKeyChecking=no -i $SSH_KEY"
CONTAINER="umbp-pd-bench"
RESULTS_BASE="$NFS_BASE/sglang/benchmark/hicache/results"

# All nodes (used by Steps 1, 3, Teardown).
ALL_NODES=( "${PREFILL_NODES[@]}" "$NODE_DECODE" )

Step 1 — Start Docker on all nodes

Loops over every prefill node plus the decode node.

for NODE in "${ALL_NODES[@]}"; do
  $SSH $NODE "
    docker rm -f $CONTAINER 2>/dev/null || true
    docker run -d --name $CONTAINER \
      --ulimit memlock=-1:-1 --ulimit stack=67108864:67108864 \
      --device /dev/dri --device /dev/kfd \
      --network host --ipc host --group-add video \
      --cap-add SYS_PTRACE --security-opt seccomp=unconfined --privileged \
      -w $NFS_BASE \
      -v /nfs:/nfs \
      -v $USER_HOME:$USER_HOME \
      $EXTRA_MOUNTS \
      --shm-size 32G \
      $DOCKER_IMAGE sleep infinity
    docker ps --filter name=$CONTAINER --format 'table {{.Names}}\t{{.Status}}'" &
done
wait
echo "Docker started on ${#ALL_NODES[@]} node(s)"

Step 2 — Start Grafana and Prometheus

Grafana and Prometheus run on the decode node. Dashboards are served from the mori and SGLang source trees on NFS.

The Prometheus config is assembled locally so all N prefill targets can be folded into one scrape job, then piped over SSH to the decode node.

# Build the multi-prefill targets list as YAML (nested under `- targets:`).
PREFILL_TARGETS=""
for ip in "${PREFILL_IPS[@]}"; do
    PREFILL_TARGETS+="          - '${ip}:30000'"$'\n'
done

cat > /tmp/pd_bench_prometheus.yml <<EOF
global:
  scrape_interval: 5s
  evaluation_interval: 30s

scrape_configs:
  - job_name: sglang_prefill
    static_configs:
      - targets:
${PREFILL_TARGETS%$'\n'}
        labels:
          role: prefill
  - job_name: sglang_decode
    static_configs:
      - targets: ['${IP_DECODE}:30001']
        labels:
          role: decode
  - job_name: umbp_master
    static_configs:
      - targets: ['${IP_PREFILL_PRIMARY}:9091']
EOF

# Push to the decode node.
$SCP /tmp/pd_bench_prometheus.yml $NODE_DECODE:/tmp/pd_bench_prometheus.yml

$SSH $NODE_DECODE "
  docker rm -f prometheus-pd 2>/dev/null || true
  docker run -d --name prometheus-pd \
    --network host \
    -v /tmp/pd_bench_prometheus.yml:/etc/prometheus/prometheus.yml:ro \
    prom/prometheus:latest \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/prometheus

  docker rm -f grafana-pd 2>/dev/null || true
  docker run -d --name grafana-pd \
    --network host \
    -v ${NFS_BASE}/sglang/examples/monitoring/grafana/datasources:/etc/grafana/provisioning/datasources:ro \
    -v ${NFS_BASE}/sglang/examples/monitoring/grafana/dashboards/config:/etc/grafana/provisioning/dashboards:ro \
    -v ${NFS_BASE}/sglang/examples/monitoring/grafana/dashboards/json:/var/lib/grafana/dashboards:ro \
    -v ${NFS_BASE}/mori/examples/monitoring/grafana/dashboards:/var/lib/grafana/mori_dashboards:ro \
    -e GF_AUTH_ANONYMOUS_ENABLED=true \
    -e GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer \
    -e GF_AUTH_BASIC_ENABLED=false \
    -e GF_USERS_ALLOW_SIGN_UP=false \
    -e GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/sglang-dashboard.json \
    grafana/grafana:latest

  sleep 4
  curl -sf http://localhost:9090/-/ready && echo 'Prometheus: READY' || echo 'Prometheus: NOT READY'
  curl -sf http://localhost:3000/api/health | python3 -c \"import sys,json; d=json.load(sys.stdin); print('Grafana:', 'HEALTHY' if d.get('database')=='ok' else 'NOT READY')\" 2>/dev/null || echo 'Grafana: NOT READY'
"
echo "=== Grafana:    http://${IP_DECODE}:3000 ==="
echo "=== Prometheus: http://${IP_DECODE}:9090 ==="

Step 3 — Build mori on all nodes

Clears the per-node build directory so cmake starts clean, then builds with UMBP enabled. Builds run in parallel across every prefill node and the decode node.

for NODE in "${ALL_NODES[@]}"; do
  $SSH $NODE "docker exec $CONTAINER bash -c '
    rm -rf ${NFS_BASE}/mori/build_\$(hostname) &&
    cd ${NFS_BASE}/mori && UMBP=ON bash build.sh'" &
done
wait
echo "mori builds done on ${#ALL_NODES[@]} node(s)"

Step 4 — Create launcher scripts

All launchers point at a single UMBP master on the primary prefill node (${IP_PREFILL_PRIMARY}:15558). Per-node behavior differs only on three variables:

Variable

Primary prefill (i=0)

Secondary prefill (i>=1)

Decode

UMBP_MASTER_AUTO_START

(default true)

false

false

UMBP_NODE_ADDRESS

${PREFILL_IPS[0]}

${PREFILL_IPS[i]}

${IP_DECODE}

--role

prefill

prefill

decode

UMBP_IO_ENGINE_PORT and UMBP_PEER_SERVICE_PORT are required whenever UMBP_MASTER_ADDRESS is set. UMBP_NODE_ADDRESS must be unique per node so all 8 * (N+1) dp ranks register distinct identities in the master.

Note on USE_DUMMY_WEIGHTS: set to true below to skip loading real model weights, which speeds up startup and is useful for benchmarking transfer throughput. Set to false (or remove the variable) to run with actual weights.

# Helper that emits a prefill launcher for a given index.
# Index 0 is primary (auto-starts the UMBP master); indices >=1 connect to it.
emit_prefill_launcher() {
    local idx="$1"
    local node_ip="${PREFILL_IPS[$idx]}"
    local out="/tmp/launch_pd_prefill_${idx}.sh"
    local extra_master_line=""
    if (( idx > 0 )); then
        extra_master_line="export UMBP_MASTER_AUTO_START=false"
    fi
    cat > "$out" << LAUNCHEOF
#!/bin/bash
export PYTHONPATH=${NFS_BASE}/mori/python:/sgl-workspace/aiter
export MC_IB_TC=96
export MORI_ENABLE_SDMA=0
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1800
export SGLANG_MORI_FP4_DISP=false
export SGLANG_MORI_FP8_DISP=true
export SGLANG_MORI_FP8_COMB=true
export SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK=2048
export NCCL_IB_HCA=${NCCL_IB_HCA}
export GLOO_SOCKET_IFNAME=${NET_IFNAME}
export NCCL_SOCKET_IFNAME=${NET_IFNAME}
export MORI_SOCKET_IFNAME=${NET_IFNAME}
export MORI_RDMA_DEVICES=${MORI_RDMA_DEVICES}
export SGLANG_USE_AITER=1
export KV_CACHE_DTYPE=fp8_e4m3
export UMBP_MASTER_ADDRESS=${IP_PREFILL_PRIMARY}:15558
${extra_master_line}
export UMBP_NODE_ADDRESS=${node_ip}
export UMBP_MASTER_BIN=${NFS_BASE}/mori/build_\$(hostname)/src/umbp/umbp_master
export UMBP_IO_ENGINE_HOST=127.0.0.1
export UMBP_IO_ENGINE_PORT=16000
export UMBP_PEER_SERVICE_PORT=16001
export UMBP_CACHE_REMOTE_FETCHES=false
export ENABLE_KV_EVENTS=true
export KV_EVENTS_PUBLISHER=zmq
export KV_EVENTS_ENDPOINT=tcp://*:5557
export KV_EVENTS_TOPIC=
export USE_DUMMY_WEIGHTS=true
export MEM_FRACTION_STATIC=0.7
export MORI_GLOBAL_LOG_LEVEL=info
export MORI_LOG_FILE=${USER_HOME}/mori_prefill_${idx}.log
exec bash ${NFS_BASE}/sglang/benchmark/hicache/run_pd_disagg_bench_dp8ep8.sh --role prefill
LAUNCHEOF
    chmod +x "$out"
}

# Build space-separated PREFILL_URLS list for the decode launcher.
# The bench script's xP1D contract: pass each prefill URL once, separated by
# a space. Bootstrap port is shared (DISAGG_BOOTSTRAP_PORT, default 8998).
PREFILL_URLS_LIST=""
for ip in "${PREFILL_IPS[@]}"; do
    PREFILL_URLS_LIST+="http://${ip}:30000 "
done
PREFILL_URLS_LIST="${PREFILL_URLS_LIST% }"   # trim trailing space

cat > /tmp/launch_pd_decode.sh << LAUNCHEOF
#!/bin/bash
export PYTHONPATH=${NFS_BASE}/mori/python:/sgl-workspace/aiter
export MC_IB_TC=96
export MORI_ENABLE_SDMA=0
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1800
export SGLANG_MORI_FP4_DISP=false
export SGLANG_MORI_FP8_DISP=true
export SGLANG_MORI_FP8_COMB=true
export SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK=2048
export NCCL_IB_HCA=${NCCL_IB_HCA}
export GLOO_SOCKET_IFNAME=${NET_IFNAME}
export NCCL_SOCKET_IFNAME=${NET_IFNAME}
export MORI_SOCKET_IFNAME=${NET_IFNAME}
export MORI_RDMA_DEVICES=${MORI_RDMA_DEVICES}
export SGLANG_USE_AITER=1
export KV_CACHE_DTYPE=fp8_e4m3
export UMBP_MASTER_ADDRESS=${IP_PREFILL_PRIMARY}:15558
export UMBP_MASTER_AUTO_START=false
export UMBP_NODE_ADDRESS=${IP_DECODE}
export UMBP_MASTER_BIN=${NFS_BASE}/mori/build_\$(hostname)/src/umbp/umbp_master
export UMBP_IO_ENGINE_HOST=127.0.0.1
export UMBP_IO_ENGINE_PORT=16000
export UMBP_PEER_SERVICE_PORT=16001
export UMBP_CACHE_REMOTE_FETCHES=false
export ENABLE_KV_EVENTS=true
export KV_EVENTS_PUBLISHER=zmq
export KV_EVENTS_ENDPOINT=tcp://*:5557
export KV_EVENTS_TOPIC=
export USE_DUMMY_WEIGHTS=true
export MEM_FRACTION_STATIC=0.7
export OUTPUT_LENGTH=${OUTPUT_LENGTH}
export PREFILL_URLS="${PREFILL_URLS_LIST}"
export MORI_GLOBAL_LOG_LEVEL=info
export MORI_LOG_FILE=${USER_HOME}/mori_decode.log
exec bash ${NFS_BASE}/sglang/benchmark/hicache/run_pd_disagg_bench_dp8ep8.sh --role decode
LAUNCHEOF
chmod +x /tmp/launch_pd_decode.sh

# Emit one prefill launcher per node and ship them.
for i in "${!PREFILL_NODES[@]}"; do
    emit_prefill_launcher "$i"
    $SCP "/tmp/launch_pd_prefill_${i}.sh" \
        "${PREFILL_NODES[$i]}:${USER_HOME}/launch_pd_prefill.sh"
done
$SCP /tmp/launch_pd_decode.sh $NODE_DECODE:$USER_HOME/launch_pd_decode.sh

Step 5 — Kill stale processes and launch prefill

Clears any leftover SGLang or UMBP master processes from a previous run on every node, then launches prefill nodes. Startup order matters:

  1. Primary first (PREFILL_NODES[0]) — its bench script auto-starts the shared UMBP master.

  2. Secondaries (PREFILL_NODES[1..N-1]) — only after the master is up, since they connect to it with UMBP_MASTER_AUTO_START=false.

For 1P1D, the secondaries loop is a no-op.

# Stale-process cleanup on all nodes (parallel).
for NODE in "${ALL_NODES[@]}"; do
  $SSH $NODE 'docker exec umbp-pd-bench bash -c "
    pkill -9 -f sglang 2>/dev/null
    pkill -9 -f umbp_master 2>/dev/null
    sleep 2
    ss -tlnp | grep -E \":30000|:30001\" || echo prefill_decode_ports_free"' &
done
wait

# Marker file used by Step 6/8 to filter out previous-run log dirs on NFS.
RUN_START_MARKER="/tmp/pd_bench_run_start"
touch "$RUN_START_MARKER"

# 5a. Launch the PRIMARY prefill (auto-starts the UMBP master).
$SSH "$NODE_PREFILL_PRIMARY" "docker exec -d $CONTAINER bash -c \
  'bash $USER_HOME/launch_pd_prefill.sh > $USER_HOME/pd_bench_prefill.log 2>&1'"
echo "Primary prefill launched on $NODE_PREFILL_PRIMARY"

# 5b. Wait for the UMBP master to be reachable, then launch secondaries.
# Secondaries fail fast if the master is not yet listening on :15558.
if (( N_PREFILLS > 1 )); then
    echo "Waiting for UMBP master at ${IP_PREFILL_PRIMARY}:15558..."
    for i in $(seq 1 60); do
        if $SSH "$NODE_PREFILL_PRIMARY" "ss -tlnp 2>/dev/null | grep -q ':15558'"; then
            echo "  UMBP master is listening (took ${i}s window)."
            break
        fi
        sleep 5
        if (( i == 60 )); then
            echo "ERROR: UMBP master did not start within 5min — abort." >&2
            exit 1
        fi
    done

    for i in $(seq 1 $((N_PREFILLS - 1))); do
        node="${PREFILL_NODES[$i]}"
        $SSH "$node" "docker exec -d $CONTAINER bash -c \
          'bash $USER_HOME/launch_pd_prefill.sh > $USER_HOME/pd_bench_prefill.log 2>&1'"
        echo "Secondary prefill launched on $node (idx $i)"
    done
fi

Step 6 — Wait for ALL prefills ready

Logs are on NFS, so they can be read directly without SSH or docker exec. For xP1D we wait until every prefill node’s server_prefill.log reports fired up. Logs from previous runs are excluded via the RUN_START_MARKER touched in Step 5.

The loop polls every 5 seconds and prints incremental output every 30 seconds.

sleep 10
declare -A LAST_LINES
for i in $(seq 1 180); do
  # Discover all server_prefill.log files newer than RUN_START_MARKER.
  mapfile -t LOGS < <(find "${RESULTS_BASE}/pd_disagg_prefill" \
                        -name server_prefill.log -newer "$RUN_START_MARKER" \
                        2>/dev/null | sort)

  # Count READY logs.
  READY_COUNT=0
  for LOG in "${LOGS[@]}"; do
    if grep -q "fired up" "$LOG" 2>/dev/null; then
      READY_COUNT=$(( READY_COUNT + 1 ))
    fi
  done

  if (( READY_COUNT >= N_PREFILLS )); then
    echo "=== All ${N_PREFILLS} prefill(s) READY (iter $i) ==="
    for LOG in "${LOGS[@]}"; do
      echo "--- $LOG (last 3) ---"
      tail -3 "$LOG"
    done
    break
  fi

  # Surface fatal errors from any prefill log.
  ERR_HIT=false
  for LOG in "${LOGS[@]}"; do
    if grep -qE "^Traceback \(most recent|^[[:space:]]*(RuntimeError|AssertionError|ImportError):|CUDA error:|out of memory|Segmentation fault|core dumped|^Killed" "$LOG" 2>/dev/null; then
      echo "=== ERROR in $LOG (iter $i) ==="
      tail -60 "$LOG"
      ERR_HIT=true
      break
    fi
  done
  $ERR_HIT && break

  # Periodic incremental tail per log.
  if (( i % 6 == 0 )); then
    echo "--- iter $i: ${READY_COUNT}/${N_PREFILLS} prefill(s) ready ---"
    for LOG in "${LOGS[@]}"; do
      TOTAL=$(wc -l < "$LOG" 2>/dev/null || echo 0)
      LAST="${LAST_LINES[$LOG]:-0}"
      if (( TOTAL > LAST )); then
        echo "  >>> $LOG (lines $LAST -> $TOTAL)"
        tail -n +"$((LAST + 1))" "$LOG" | head -10
        LAST_LINES[$LOG]=$TOTAL
      fi
    done
  fi

  # Wrapper logs on every prefill node — surface FATAL/FAILED early.
  WRAP_HIT=false
  for NODE in "${PREFILL_NODES[@]}"; do
    WLOG=$($SSH "$NODE" "cat $USER_HOME/pd_bench_prefill.log 2>/dev/null" || true)
    if echo "$WLOG" | grep -qiE "^\[.*\] FATAL|^\[.*\] FAILED"; then
      echo "=== FATAL in wrapper log on $NODE ==="
      echo "$WLOG" | tail -20
      WRAP_HIT=true
      break
    fi
  done
  $WRAP_HIT && break

  sleep 5
done

Step 7 — Launch decode and benchmark

Decode connects to the already-running UMBP master on the primary prefill node, and the bench script’s sglang_router fans out to all N prefills via PREFILL_URLS. The wrapper waits for all N+1 workers to register on the router before starting the bench (timeout: ROUTER_WORKER_READY_TIMEOUT_SECS, default 180s).

$SSH $NODE_DECODE "docker exec -d $CONTAINER bash -c \
  'bash $USER_HOME/launch_pd_decode.sh > $USER_HOME/pd_bench_decode.log 2>&1'"
echo "Decode + benchmark launched (router will fan out to ${N_PREFILLS} prefill(s))"

Step 8 — Monitor decode and benchmark

Polls every 15 seconds. Prints incremental server log every 60 seconds. Exits when the benchmark reports completion or a fatal error is detected.

sleep 10
DECODE_READY=false
LAST=0
for i in $(seq 1 720); do
  # Filter out previous-run decode logs via RUN_START_MARKER from Step 5.
  LOG=$(find ${RESULTS_BASE}/pd_disagg_decode -name server_decode.log \
          -newer "$RUN_START_MARKER" 2>/dev/null | sort -r | head -1)
  if [[ -n "$LOG" ]]; then
    if ! $DECODE_READY && grep -q "fired up" "$LOG"; then
      echo "--- Decode READY (iter $i), benchmark starting ---"
      DECODE_READY=true
    fi
    if grep -qE "^Traceback \(most recent|^[[:space:]]*(RuntimeError|AssertionError|ImportError):|CUDA error:|out of memory|Segmentation fault|core dumped|^Killed" "$LOG"; then
      echo "=== ERROR in decode log (iter $i) ===" && tail -60 "$LOG" && break
    fi
    TOTAL=$(wc -l < "$LOG")
    if (( i % 4 == 0 && TOTAL > LAST )); then
      echo "--- decode log update (iter $i, lines $LAST→$TOTAL) ---"
      tail -n +"$((LAST + 1))" "$LOG" | head -15
      LAST=$TOTAL
    fi
  fi
  WLOG=$($SSH $NODE_DECODE "cat $USER_HOME/pd_bench_decode.log 2>/dev/null" || true)
  if echo "$WLOG" | grep -q "Benchmark finished in"; then
    echo "=== Benchmark COMPLETE (iter $i) ===" && echo "$WLOG" | tail -20 && break
  fi
  if echo "$WLOG" | grep -qiE "^\[.*\] FATAL|SERVER_CRASH|^\[.*\] FAILED"; then
    echo "=== FATAL in decode wrapper (iter $i) ===" && echo "$WLOG" | tail -30
    [[ -n "$LOG" ]] && tail -30 "$LOG"
    break
  fi
  sleep 15
done

Step 9 — Show results

For xP1D, all N prefill logs are listed (one per node). The RUN_START_MARKER filter ensures we only see this run’s logs.

echo "=== Prefill server logs (this run, last 20 each) ==="
mapfile -t PLOGS < <(find ${RESULTS_BASE}/pd_disagg_prefill \
                       -name server_prefill.log -newer "$RUN_START_MARKER" \
                       2>/dev/null | sort)
for LOG in "${PLOGS[@]}"; do
  echo "--- $LOG ---"
  tail -20 "$LOG"
done

echo "=== Decode server log (last 20) ==="
find ${RESULTS_BASE}/pd_disagg_decode -name server_decode.log \
    -newer "$RUN_START_MARKER" 2>/dev/null | sort -r | head -1 | xargs tail -20

echo "=== Decode wrapper log (last 20) ==="
$SSH $NODE_DECODE "tail -20 $USER_HOME/pd_bench_decode.log"

echo "=== Summary ==="
find ${RESULTS_BASE}/pd_disagg_decode -name summary.txt \
    -newer "$RUN_START_MARKER" 2>/dev/null | sort -r | head -1 | xargs cat 2>/dev/null

echo "=== Metrics ==="
find ${RESULTS_BASE}/pd_disagg_decode -name performance_metrics.jsonl \
    -newer "$RUN_START_MARKER" 2>/dev/null | sort -r | head -1 | xargs tail -5 2>/dev/null

Teardown

for NODE in "${ALL_NODES[@]}"; do
  $SSH $NODE "
    docker exec $CONTAINER bash -c 'pkill -9 -f sglang; pkill -9 -f umbp_master' 2>/dev/null || true
    docker rm -f $CONTAINER" &
done
$SSH $NODE_DECODE "docker rm -f prometheus-pd grafana-pd 2>/dev/null || true" &
wait

Environment Variable Reference

Topology (this guide’s wrapper variables)

Variable

Description

PREFILL_NODES

Bash array of prefill hostnames; index 0 is the primary (hosts the UMBP master). Length determines N in xP1D.

PREFILL_IPS

Bash array of prefill IPs; same length and order as PREFILL_NODES.

NODE_PREFILL_PRIMARY

Derived: ${PREFILL_NODES[0]}.

IP_PREFILL_PRIMARY

Derived: ${PREFILL_IPS[0]}; UMBP master listens on ${IP_PREFILL_PRIMARY}:15558.

N_PREFILLS

Derived: ${#PREFILL_NODES[@]}.

NODE_DECODE / IP_DECODE

The single decode node.

ALL_NODES

Derived: ("${PREFILL_NODES[@]}" "$NODE_DECODE").

UMBP / mori (consumed by run_pd_disagg_bench_dp8ep8.sh)

Variable

Description

UMBP_MASTER_ADDRESS

host:port of the shared UMBP master (always ${IP_PREFILL_PRIMARY}:15558)

UMBP_MASTER_AUTO_START

true on the primary prefill (default); false on every secondary prefill and on decode

UMBP_NODE_ADDRESS

This node’s IP — must be unique per node so all 8 * (N+1) dp ranks register distinct identities in the master

UMBP_MASTER_BIN

Path to the umbp_master binary (per-node build directory)

UMBP_IO_ENGINE_HOST

Host for the local IO engine listener (always 127.0.0.1)

UMBP_IO_ENGINE_PORT

Port for the local IO engine; required when UMBP_MASTER_ADDRESS is set

UMBP_PEER_SERVICE_PORT

Port for peer-to-peer service; required when UMBP_MASTER_ADDRESS is set

UMBP_CACHE_REMOTE_FETCHES

Set false to disable remote fetch caching during benchmarking

SGLANG_MORI_FP8_DISP

Enable FP8 dispatch (reduces transfer size)

SGLANG_MORI_FP8_COMB

Enable FP8 combine

MORI_RDMA_DEVICES

RDMA devices to use; prefix ^ to exclude (e.g., ^mlx5_8)

MORI_SOCKET_IFNAME

Network interface for mori’s TCP socket

KV_CACHE_DTYPE

KV cache dtype; fp8_e4m3 recommended for transfer efficiency

KV_EVENTS_PUBLISHER

KV event publisher backend (zmq)

KV_EVENTS_ENDPOINT

ZMQ endpoint for KV events (e.g., tcp://*:5557)

ENABLE_KV_EVENTS

Enable KV event publishing

USE_DUMMY_WEIGHTS

Skip loading real model weights (for throughput benchmarking)

MEM_FRACTION_STATIC

Fraction of GPU memory reserved for static KV cache

OUTPUT_LENGTH

Number of decode output tokens per request (decode node only)

PREFILL_URLS

Space-separated prefill URLs for the decode node’s sglang_router (e.g. "http://10.0.0.1:30000 http://10.0.0.2:30000"); the bench script fans out one --prefill per entry

PREFILL_URL

[DEPRECATED] Single-prefill shortcut (1P1D back-compat); folded into PREFILL_URLS if set

ROUTER_WORKER_READY_TIMEOUT_SECS

Max seconds the wrapper waits for all N+1 workers to report load >= 0 on /get_loads after the router is ready. Default 180.

Troubleshooting

SSH access denied on prefill node from local machine

Use the decode node as a jumphost to authorize your key on every prefill node.

PUBKEY=$(cat $USER_HOME/.ssh/id_ed25519.pub)
for NODE in "${PREFILL_NODES[@]}"; do
  $SSH $NODE_DECODE "ssh -o StrictHostKeyChecking=no $NODE \
    \"mkdir -p ~/.ssh && echo '$PUBKEY' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys\""
done

Server fails to start / ImportError

Check PYTHONPATH includes both the mori Python bindings and aiter:

export PYTHONPATH=${NFS_BASE}/mori/python:/sgl-workspace/aiter

The NFS copy of mori (${NFS_BASE}/mori/python) is authoritative. The docker-internal /sgl-workspace/mori may be an older version and should not be used.

Errors appear in the wrong log

Always check the actual SGLang server log, not the wrapper log:

# Prefill
find ${RESULTS_BASE}/pd_disagg_prefill -name server_prefill.log | sort -r | head -1 | xargs tail -50

# Decode
find ${RESULTS_BASE}/pd_disagg_decode -name server_decode.log | sort -r | head -1 | xargs tail -50

Port already in use

Kill stale processes from a previous run (see Step 5) and verify the port is free. For xP1D, only the primary prefill is expected to be listening on :15558 (UMBP master); secondary prefills should not be (they connect to the primary).

for NODE in "${PREFILL_NODES[@]}"; do
  echo "--- $NODE ---"
  $SSH $NODE "ss -tlnp | grep -E ':30000|:15558|:16000|:16001' || echo all_free"
done
echo "--- $NODE_DECODE ---"
$SSH $NODE_DECODE "ss -tlnp | grep -E ':30001|:16000|:16001|:8000' || echo all_free"

Bench progress freezes partway

If bench_multiturn sits at e.g. 4/8 for minutes with no new requests, check the decode wrapper log for WARNING: router workers not all healthy. That means ROUTER_WORKER_READY_TIMEOUT_SECS (default 180s) elapsed before all N+1 workers registered, and the bench then raced a 503 "No decode workers available". Increase the timeout and re-run:

$SSH $NODE_DECODE "docker exec $CONTAINER bash -c \
  'ROUTER_WORKER_READY_TIMEOUT_SECS=300 bash $USER_HOME/launch_pd_decode.sh > $USER_HOME/pd_bench_decode.log 2>&1'"