madengine — Codebase Wiki

Overview

Where to start

Quick start

Run these from a MAD-style project directory (one that contains models.json and/or scripts/). Every flag and key below is documented in the CLI and configuration sections.

# 1. Install (from the madengine repo checkout)
pip install -e .

# 2. List the cards a tag selects
madengine discover --tags dummy

# 3. Build and run in one command
madengine run --tags dummy \
  --additional-context '{"gpu_vendor":"AMD","guest_os":"UBUNTU"}'

# 4. Or build first (optionally push), then run from the manifest
madengine build --tags dummy --registry registry.example.com/ml
madengine run --manifest-file build_manifest.json \
  --additional-context '{"docker_gpus":"0,1,2,3"}'

madengine run --tags my_model --live-output \
  --additional-context '{
    "platform": "bare_metal",
    "conda": {"conda_bin": "/path/to/micromamba"}
  }'

madengine build --tags llama3 --registry registry.example.com/ml

# Multi-node torchrun via sbatch
madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {"partition":"gpu","nodes":4,"gpus_per_node":8,"time":"24:00:00"},
    "distributed": {"launcher":"torchrun","nnodes":4,"nproc_per_node":8}
  }'

# A card that drives its own per-node containers (slurm_multi):
# skip the local build; "auto" reads DOCKER_IMAGE_NAME from the card's env_vars
madengine build --tags pyt_sglang_disagg --use-image auto

madengine run --tags llama3 \
  --additional-context '{"k8s":{"namespace":"ml-team","gpu_count":4}}'

# Attach mode: benchmark an llm-d stack that already exists
madengine run --tags dummy_llm_d --additional-context '{
  "k8s":   {"namespace":"llm-d-bench","gpu_count":0},
  "llm_d": {
    "endpoint_url":"http://llm-d-inference-gateway.llm-d-bench.svc.cluster.local:80",
    "model":{"name":"Qwen/Qwen3-32B"}
  }
}'

# Context file (JSON object)
madengine run --tags llama3 --additional-context-file my_run.json

# Hydra / YAML frontend (same keys as the context, plus config groups)
madengine run --config my_job.yaml
madengine run --config my_job.yaml --config distributed.nnodes=4
madengine run --config my_job.yaml --config +env=nccl_debug

Install & dev

git clone https://github.com/ROCm/madengine.git
cd madengine
python3 -m venv venv && source venv/bin/activate

# Editable install: ALL runtime and dev dependencies come with it
pip install -e .

# Optional: host-side TraceLens analysis (see "madengine report tracelens")
pip install -e '.[tracelens]'

pre-commit install
madengine --version

Dependencies and extras

Everything is declared in pyproject.toml under [project] dependencies, including the Kubernetes client, Hydra, MongoDB, and the dev toolchain. There is exactly one optional extra:

Versioning

The version is not stored in the source tree. versioningit derives it from git tags matching v<version> (default tag v2.0.0 when none exists). Untagged commits become <base>.post<distance>+g<sha>; a dirty tree adds a build date (<base>+d<YYYYMMDD> on a tag, …+g<sha>.d<YYYYMMDD> after it). madengine --version and madengine.__version__ read the installed package metadata, so an uninstalled checkout reports unknown / dev.

pytest                           # testpaths: tests/unit, integration, e2e
pytest tests/unit -v             # fast, mocked
pytest tests/unit/test_slurm_multi.py -v
pytest -k llm_d                  # name filter
pytest -m "not slow"             # skip slow tests
pytest -m "amd or cpu"           # hardware markers
pytest --cov=src/madengine --cov-report=html

black src/ tests/
isort src/ tests/
flake8 src/ tests/
mypy src/madengine
pre-commit run --all-files

Layered architecture

Each layer calls only the layers below it. Layers are color-coded throughout this page.

Packaged assets live beside the code: src/madengine/scripts/ (pre/post scripts, tools, SLURM/K8s helper scripts) is copied into a project's scripts/common/ for a run and removed afterwards. See Module reference.

Architecture diagram

Key data flows

Derived from orchestration/build_orchestrator.py and orchestration/run_orchestrator.py. Both orchestrators receive a single merged additional_context dict from the CLI (see additional_context).

CLI overview

Entry point: src/madengine/cli/app.py::cli_main, installed as the madengine console script. The Typer app registers seven top-level commands; report is a sub-app with its own sub-commands.

Global behavior

• madengine --version prints the installed package version (importlib.metadata; unknown if not installed). With no sub-command the help is shown. Shell completion is disabled (add_completion=False).
• cli_main() first calls core.lifecycle.install_signal_handlers() so SIGINT/SIGTERM/SIGHUP tear down benchmark containers (see container lifecycle). A KeyboardInterrupt or an unhandled exception that escapes a command exits 1.
• A Rich traceback handler is installed with show_locals=True. An unhandled crash can therefore print local variable values; do not rely on the console log being free of credentials.

Context sources

build and run accept the same configuration spine through three mutually exclusive-in-part inputs:

Validation applied to the merged context

Implemented in cli/validators.py::finalize_additional_context_dict; any failure exits 4 (INVALID_ARGS). build always validates; run validates only when a non-empty context was supplied.

1. Missing gpu_vendor / guest_os are defaulted to AMD / UBUNTU (core/additional_context_defaults.py) and a banner says so.
2. Structure checks: docker_build_arg, docker_env_vars, docker_mounts, env_vars, k8s, slurm, kubernetes, llm_d, distributed, vllm, deployment_config, deploy must be objects; tools and pre_scripts arrays; docker_gpus and MAD_CONTAINER_IMAGE strings; timeout a number; debug a boolean; log_error_patterns a non-empty string array. Build-arg values are coerced to strings.
3. gpu_vendor must be AMD or NVIDIA, guest_os must be UBUNTU or CENTOS (case-insensitive, normalized to upper case).
4. Launcher: every distributed.launcher / launcher.type is validated and rewritten to its canonical spelling. An unknown name is rejected up front rather than running as a single process. A bare-string top-level launcher is rejected. The per-target rule (slurm_multi is not valid on llm-d) is enforced when the deployment is constructed. See Launchers.
5. Platform: platform is a string; omitted means Docker, "bare_metal" selects conda. Unknown values, the retired top-level bare_metal object, and a conda block on a Docker run are rejected. See Platforms.

Common option letters

How failures become exit codes

Each command wraps its body and maps exception classes from core/errors.py to an exit code. The mapping is not identical across commands:

CLI — discover

Lists model cards and validates tag selection without building or running anything. Must be run from the MAD project root: discovery reads ./models.json (required; FileNotFoundError otherwise) and walks ./scripts/.

Tag syntax

Discovery sources

1. Root ./models.json — required.
2. scripts/**/models.json — found by a recursive walk, so nested submodule layouts (scripts/MAD/…) work.
3. scripts/**/get_models_json.py — dynamic; must define list_models() returning CustomModel objects (utils/discover_models.py).

A directory containing both models.json and get_models_json.py is an error. If MODEL_DIR points elsewhere than the current directory, discovery first copies docker/, scripts/, models.json, credential.json and data.json from it into the working directory.

CLI — build

Builds Docker images for the selected models (or creates conda environments when platform is bare_metal) and writes build_manifest.json. Context is always validated; gpu_vendor / guest_os default to AMD / UBUNTU when absent.

Self-managed launchers need an image source

For a card whose launcher is slurm_multi, the build requires one of: --registry, --use-image, --build-on-compute --registry, or a single env_vars.DOCKER_IMAGE_NAME on the card(s). In the last case, with no registry given, it is used as an implicit --use-image. Otherwise the build raises ConfigurationError (exit 4). An angle-bracket placeholder such as <your-image> in DOCKER_IMAGE_NAME is rejected.

Exit status

0 only if every build succeeded and every push succeeded. Any failure, including a partial one, exits 2; images that did build remain in the manifest.

CLI — run

Runs models and writes perf.csv. Mode is chosen by the manifest:

• --manifest-file names an existing file → execution only (--tags is not used to select models).
• Otherwise → full workflow: build, then run. A named-but-missing manifest only prints a warning. With neither a manifest nor --tags (and no MAD_CONTAINER_IMAGE) the orchestrator raises ConfigurationError (exit 4).
• MAD_CONTAINER_IMAGE in context skips the build and synthesizes a manifest for the given image; --tags is then required.

Timeout resolution

Single definition in core/timeout.py::resolve_run_timeout, shared by local, SLURM and Kubernetes runs. Precedence, lowest to highest: default (DEFAULT_RUN_TIMEOUT = 7200 s) < model card timeout < explicit --timeout.

A card may itself set "timeout" to 0 or any non-positive value to mean "unbounded"; null in an old manifest is ignored. Non-positive resolved values are mapped to "no timeout" for subprocess by subprocess_timeout() (since subprocess would treat 0 as "expire immediately").

Exit status

0 when every model succeeded; 3 if any model failed; 2 when every build of a full workflow failed (BuildError); 4 for invalid arguments/configuration; 1 otherwise. A full workflow whose build only partly failed still runs the images that built; if those runs all pass it exits 1 ("Workflow failed for unknown reasons"), not 2. After the run, run prints the per-model results table and the perf.csv table (current-session rows highlighted via utils/session_tracker.py).

CLI — report & database

madengine report

Sub-app (report_app, no_args_is_help=True). All sub-commands exit 1 on failure.

madengine report to-html --csv-file-path perf.csv
madengine report to-email --dir ./results -o summary.html
madengine report tracelens --discover-only
madengine report tracelens --root slurm_results --mode collective --world-size 8
madengine report tracelens-compare base.xlsx cand.xlsx --names before --names after -o diff.xlsx

madengine database

Uploads a CSV or JSON results file to MongoDB (database/mongodb.py); the format is auto-detected.

Connection comes from the environment (MongoDBConfig.from_env()):

Exit 0 for a successful upload, or a partial one that inserted/updated at least one document; 1 otherwise (missing file, connection failure, nothing written). See also Database and Reporting.

CLI — cleanup & setup-page

madengine cleanup

Removes benchmark containers left behind by runs that died before they could clean up. Every container madengine starts carries the labels madengine.session and madengine.owner_pid (core/lifecycle.py); cleanup lists containers (all states) with the session label and acts on those whose owner PID is no longer alive. It is safe to call from an if: always() CI step or from cron while other benchmarks are running, because containers with a live owner are skipped.

Removal escalates docker stop → kill → rm -f → direct SIGKILL of the container's host PIDs. See Container lifecycle for the heartbeat watchdog and signal handlers.

madengine setup-page

Generates a single, fully self-contained HTML page (no external assets, so it can be published to GitHub Pages) from the current directory's models.json. The page lists models and tags plus the madengine context schema (setup_page/schema.py::CONTEXT_SCHEMA: general, deployment target, container, distributed, Kubernetes, SLURM, tools & scripts, logging, run flags) and shows the exact madengine run command to copy. Only values that differ from defaults and match the selected deployment target are emitted.

Model collection uses DiscoverModels (so scripts/**/models.json and get_models_json.py models appear), falling back to the root models.json alone if full discovery fails. Missing models.json exits 1. The schema is data-driven: adding a context key to the page is a one-entry change in schema.py, validated by validate_schema() at render time.

--config — YAML / Hydra frontend

An alternative to --additional-context for build and run. It composes a config with Hydra's Compose API, validates it, and translates it into the same additional_context dict the rest of the pipeline already consumes, so behavior is identical to the JSON form. Code: config/loader.py (HydraConfigLoader), config/schema.py (ConfigValidator), config/translator.py (ConfigTranslator), package data in src/madengine/configs/ (shipped in the wheel).

Argument forms

--config is repeatable; each value is one of:

Precedence (later wins)

1. Package defaults: configs/config.yaml with default groups platform=docker, scheduler=local, hardware=amd, launcher=none.
2. The user YAML file, merged over the defaults.
3. Group selections and +group appends. Only values that differ from the package defaults are overlaid, so unrelated keys in your YAML are not reset. Corollary: selecting a group's default (--config scheduler=local) is a no-op.
4. key=value overrides.

Dicts merge key by key; lists are replaced (+tools=… replaces a YAML tools list, while +env=… adds to env_vars). Explicit CLI flags win over config-derived values wherever the flag was changed from its default.

Groups shipped in configs/

Group files do not cover every deployment target: there is no group for spur or llm_d, and llm_d / conda are not accepted top-level YAML keys (see validation below). Use --additional-context for those.

Job YAML shape

A job file uses the same top-level keys as the JSON context (slurm, k8s, distributed, env_vars, tools, …), not a Hydra defaults: list. Reference templates and runnable demos live in examples/configs/templates/ and examples/configs/demo/{local,slurm,k8s}/. Defaults provided by config.yaml:

Validation and restrictions

• Unknown top-level keys are errors (Unknown config key: '…'). Allowed: defaults, platform, scheduler, hardware, launcher, model, docker, build, env_vars, debug, live_output, log_error, tools, pre_scripts, post_scripts, encapsulate_script, data_config, output, summary_output, gpu_vendor, guest_os, runtime, slurm, k8s, kubernetes, distributed, vllm, sglang_disagg, shared_data, timeout, gpu_type, gpu_memory_gb, gpus_per_node, data, require_pinned_image.
• slurm together with k8s/kubernetes is rejected.
• distributed.enabled: true requires distributed.launcher; distributed.nnodes must be a positive integer.
• Platform: the object form platform: {type: …} accepts only docker; bare_metal, podman and singularity are rejected as "not yet supported", so the platform=bare_metal group fails validation. A top-level platform value is CLI metadata and is not forwarded into additional_context, and conda is not an accepted key, so bare metal cannot be selected through --config: pass "platform": "bare_metal" (and any conda block) with --additional-context.
• Any validation problem is raised as ConfigurationError listing every error (exit 4).

Launcher handling

• The launcher top-level key as a string is the Hydra group selector and is dropped from the context. As an object (type, master_port, …) it is kept as launcher for Kubernetes.
• distributed.launcher is canonicalized by the same function the engine uses elsewhere (sglang_disagg → sglang-disagg, slurm-multi → slurm_multi). The group launcher=megatron yields megatron-lm; the bare string megatron is not a valid launcher name.
• distributed.enabled is passed through as written; the translator never sets it. Every launcher group writes enabled: true, and the default launcher=none writes enabled: false. Validation requires a launcher when it is true. Kubernetes profile selection treats either enabled: true or a launcher name as distributed.
• launcher=native (and docker) is a perf.csv reporting value, not a launcher. The translator forces distributed.enabled: false and launcher: null so a Kubernetes profile cannot inject torchrun. Do not copy that group into JSON.

Examples

# Job file from the repo, scaled up and with NCCL debug env added
madengine run \
  --config examples/configs/demo/slurm/multi-node-torchrun.yaml \
  --config slurm.nodes=4 \
  --config distributed.nnodes=4 \
  --config +env=nccl_debug

# Local single GPU straight from a demo file
madengine run --config examples/configs/demo/local/single-gpu.yaml

# No YAML: compose from groups, then tweak values
madengine run \
  --config scheduler=slurm \
  --config launcher=torchrun \
  --config hardware=amd \
  --config +profile=mi300x_8gpu \
  --config distributed.nnodes=2 \
  --config 'model.tags=[dummy_torchrun]'

Exit codes CI contract

Defined in src/madengine/cli/constants.py::ExitCode (an IntEnum). Use them in CI instead of scraping logs.

additional_context — configuration spine

How it is parsed

What Context adds

core/context.py merges user values over detected ones; user-provided values are never overridden by detection.

Key reference

Top-level keys read anywhere under src/madengine. Scheduler sub-keys are documented in their own sections.

Model definition — models.json

Cards come from root models.json, scripts/<dir>/models.json, or scripts/<dir>/get_models_json.py (returns CustomModel objects). The CustomModel dataclass in utils/discover_models.py lists the typed fields; static JSON cards may carry additional keys, which are copied verbatim into the manifest built_models entry.

{
  "name": "pyt_llama3_train",
  "dockerfile": "docker/pyt_llama3",
  "dockercontext": "",
  "scripts": "scripts/llama3/run.sh",
  "url": "",
  "cred": "",
  "owner": "ml-team@example.com",
  "data": "",
  "n_gpus": "8",
  "timeout": 14400,
  "training_precision": "bf16",
  "tags": ["pyt", "training"],
  "args": "--batch-size 4",
  "multiple_results": "",
  "skip_gpu_arch": "gfx908,gfx1100",
  "docker_build_arg": {"VLLM_REF": "v0.9.0"},
  "additional_docker_run_options": "--shm-size 64g",
  "env_vars": {"MY_FLAG": "1"},
  "distributed": {"launcher": "torchrun", "nnodes": 2, "nproc_per_node": 8},
  "slurm": {"partition": "gpu", "nodes": 2, "time": "04:00:00"}
}

Field reference

Build manifest — build_manifest.json

Written by madengine build (or by a full run --tags), consumed by madengine run --manifest-file. The run phase rewrites it (filtering, merged context, docker_run_cmd).

{
  "built_images": {
    "ci-pyt_llama3_train_pyt_llama3.ubuntu.amd": {
      "model": "pyt_llama3_train",
      "docker_image": "ci-pyt_llama3_train_pyt_llama3.ubuntu.amd",
      "dockerfile": "docker/pyt_llama3.ubuntu.amd.Dockerfile",
      "base_docker": "rocm/pytorch:latest",
      "docker_sha": "sha256:<base image digest>",
      "build_duration": 183.4,
      "build_command": "docker build --network=host -t ... (secrets redacted)",
      "log_file": "pyt_llama3_train_pyt_llama3.ubuntu.amd.build.live.log",
      "gpu_vendor": "AMD",
      "gpu_architecture": "gfx942",
      "image_size_bytes": 23456789012,
      "registry": "registry.example.com/ml",
      "registry_image": "registry.example.com/ml/ci-pyt_llama3_train_pyt_llama3.ubuntu.amd",
      "image_digest": "sha256:<pushed digest>"
    }
  },
  "built_models": {
    "ci-pyt_llama3_train_pyt_llama3.ubuntu.amd": {
      "name": "pyt_llama3_train", "scripts": "scripts/llama3/run.sh", "n_gpus": "8",
      "env_vars": {"DOCKER_IMAGE_NAME": "registry.example.com/ml/ci-pyt_llama3_train_pyt_llama3.ubuntu.amd"}
    }
  },
  "context": {
    "docker_env_vars": {}, "docker_mounts": {}, "docker_build_arg": {},
    "gpu_vendor": "AMD", "guest_os": "UBUNTU", "docker_gpus": ""
  },
  "credentials_required": [],
  "summary": {
    "successful_builds": [], "failed_builds": [], "total_build_time": 183.4,
    "successful_pushes": [], "failed_pushes": []
  },
  "deployment_config": {
    "target": "slurm",
    "slurm": {"partition": "gpu", "nodes": 2},
    "distributed": {"launcher": "torchrun", "nnodes": 2, "nproc_per_node": 8},
    "env_vars": {}, "debug": false,
    "_explicit_slurm_keys": ["nodes", "partition"]
  },
  "docker_run_cmd": "docker run -t -d ... (written after a local run)"
}

Structure

Execution platforms

The platform context key chooses where the workload's dependencies live; the scheduler blocks (slurm, k8s, llm_d) choose where it runs. Omitted platform means Docker.

madengine run --tags my_model --live-output \
  --additional-context '{
    "platform": "bare_metal",
    "conda": {"conda_bin": "/path/to/micromamba"}
  }'

Self-managed launchers (local)

A card whose launcher is self-managed (slurm_multi) is not run in a madengine container: ContainerRunner._run_self_managed() executes the script directly on the host with cwd set to the script directory. Env vars from the card and from additional_context.env_vars are applied; only their names are logged. See slurm_multi.

Local launcher command (MAD_MULTI_NODE_RUNNER)

Launcher validation (what names are accepted at the config boundary) is documented in Launcher matrix. A pre-set docker_env_vars.MAD_MULTI_NODE_RUNNER is never overwritten.

Result status and log scanning

1. Performance is extracted from the log with the pattern performance: <number> <metric> (fallback: HuggingFace train_samples_per_second), or from the multiple_results CSV.
2. Unless log_error_pattern_scan is false, the log is scanned for substrings: OutOfMemoryError, HIP out of memory, CUDA out of memory, RuntimeError:, AssertionError:, ValueError:, SystemExit, failed (exitcode:, Traceback (most recent call last), FAILED, Exception:, ImportError:, ModuleNotFoundError: (replaceable via log_error_patterns). Known-benign lines (metrics-agent noise, FutureWarning, rocprof/rpd tracer lines, plus log_error_benign_patterns) are ignored.
3. Status: valid performance → SUCCESS even if an error pattern matched (model output can legitimately contain ValueError:); error pattern and no performance → FAILURE; worker nodes (MAD_COLLECT_METRICS=false) and skip_perf_collection runs are not failed for lacking local performance; otherwise no performance → FAILURE. A timeout, a setup exception or a non-zero script exit records a FAILURE row so the results table stays complete.

Data providers

core/dataprovider.py: data.json (plus context data) maps a data name to provider blocks. Providers are tried in the order custom, local, minio, nas, aws. --force-mirror-local <path> sets mirrorlocal to that path on every non-local provider so data is rsynced instead of SSHFS-mounted (keeping the container unprivileged). Selected provider name, type, size and download time are recorded on the result row.

Container lifecycle & cancellation

Implemented in core/lifecycle.py and core/docker.py. Problem solved: a cancelled CI job kills madengine's process tree, but the detached benchmark container survives, keeps /dev/kfd open and blocks the next job on the GPU. Three layers, most to least robust:

Labels, registry and tuning

Docker build

Implemented in execution/docker_builder.py (DockerBuilder); orchestrated by BuildOrchestrator (see flows).

docker build [--no-cache] --network=host \
  [--build-context tools=docker/common] \
  -t <tag> --pull -f <dockerfile> \
  <--build-arg ...> <context>

GPU architecture handling

Push, digest and manifest

• docker tag + docker push (no timeout). A push failure marks the entry push_failed and does not fail the build; it is listed in push_failures and summary.failed_pushes.
• The pushed digest is parsed from docker push output (fallback docker image inspect RepoDigests) and stored as image_digest (core/image_digest.py). It is only used with --require-pinned-image: the run pulls repo@sha256:... and refuses a manifest without a digest rather than falling back to the tag.
• After a successful push env_vars.DOCKER_IMAGE_NAME on the built model is set to the registry image so slurm_multi nodes can pull in parallel.
• image_size_bytes is recorded so the SLURM node health check can compare it with free disk on each candidate node.
• The run side labels locally rebuilt images with mad.build_fingerprint (Dockerfile hash + build args) and rebuilds when a reused tag no longer matches; MAD_DOCKER_BUILDS may point at a shared directory for docker save tars that other SLURM nodes docker load.

Alternative build modes

Deployment target inference

There is no required deploy field. RunOrchestrator._infer_deployment_target() inspects the merged context (runtime --additional-context over the manifest's deployment_config) and picks the first rule that matches.

Precedence

Build-time validation

BuildOrchestrator runs ConfigLoader.load_config(), which calls infer_and_validate_deploy_type() and applies the SLURM or K8s presets. It raises (surfaced as ConfigurationError) when:

• k8s/kubernetes and slurm are both present;
• llm_d and slurm are both present;
• an explicit deploy value contradicts the blocks: k8s/kubernetes without a k8s block, slurm/spur without a slurm block, spur with a slurm.scheduler other than spur, or local with a scheduler block;
• slurm.scheduler is anything other than slurm or spur.

Only k8s, slurm and spur get preset defaults here. An llm_d config with a k8s block gets the K8s presets; the llm-d defaults are applied later by LlmdDeployment itself.

Factory registration

deployment/factory.py registers deployments at import time. DeploymentFactory.create() raises ValueError listing the registered names for an unknown target.

local, docker and bare_metal are not factory targets: RunOrchestrator runs them directly.

Launcher validity per target

Every launcher in the context, the manifest deployment_config and each model card is checked when a BaseDeployment is constructed (_validate_launchers()), so a bad value surfaces as INVALID_ARGS rather than a failed job. Unknown spellings raise ConfigurationError; they do not fall back to a single process. ensure_launcher_allowed() (deployment/common.py) has a single per-target rule:

sglang-disagg node minimums (2 on SLURM/Spur, 3 on Kubernetes) are checked when the launcher command is generated, before submission; see Per-launcher configuration.

Canonical names and aliases are listed in Launchers.

SLURM & Spur deployment

Implemented by SlurmDeployment in deployment/slurm.py (node preflight in slurm_node_selector.py) and its subclass SpurDeployment in spur.py. madengine drives the scheduler through its local CLI (sbatch, squeue, sacct, scontrol, srun, scancel): run it on a login node; no SSH and no Python SLURM library are involved. The target is chosen by the presence of a slurm key (see Deployment target inference). The sbatch script is rendered from deployment/templates/slurm/job.sh.j2.

Selecting and validating the target

Checks live in ConfigLoader.infer_and_validate_deploy_type() (deployment/config_loader.py). Launcher validation runs in BaseDeployment.__init__ so a bad launcher exits with INVALID_ARGS before anything is submitted; see Launcher matrix. validate() requires sbatch, squeue, scontrol (Spur: sbatch, squeue, sacct) and a working sinfo -h.

Preset merge order

ConfigLoader.load_slurm_config() deep-merges three layers (last wins). Nested dicts merge key by key; lists and scalars are replaced; env_vars merges per variable.

1. presets/slurm/defaults.json
2. One profile, chosen by the merged slurm.nodes: profiles/multi-node.json when nodes > 1, otherwise profiles/single-node.json
3. Your --additional-context / --additional-context-file / --config result

The loader also records _preset_env_keys: the env_vars keys that only a preset supplied. The self-managed slurm_multi path drops those keys (the card's own script owns its environment); a key you set yourself is always kept.

slurm sub-keys

Every key below is read in code. Defaults are the effective value after presets.

Related top-level keys read by the SLURM path

What the job script does (templated launchers)

1. Header. #SBATCH --job-name=madengine-<model> (a / in the card name becomes _), --output/--error=<output_dir>/madengine-<model>_%j_%t.{out,err}, --nodes N --ntasks N --ntasks-per-node=1, --gpus-per-node, --time, then the optional directives from the table above.
2. Distributed environment. MASTER_ADDR is the first host of SLURM_NODELIST; MASTER_PORT, WORLD_SIZE (= SLURM_NTASKS), LOCAL_RANK, NNODES, GPUS_PER_NODE. GPU visibility: HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES for ordinary launchers; for the Ray-based vllm, sglang and sglang-disagg only the one matching the detected vendor is set, and RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES is unset.
3. Single node. Uses the submission directory when it is on nfs*, lustre, gpfs, ceph, beegfs or panfs (judged from the df filesystem-type column only); otherwise a job-scoped local workspace under $SLURM_TMPDIR or /tmp. It writes build_manifest_local.json with deployment_config.target="docker" and the slurm/k8s blocks removed, exports MAD_MULTI_NODE_RUNNER="torchrun --standalone --nproc_per_node=<gpus>", and runs madengine run --manifest-file build_manifest_local.json --timeout <cli value> locally on the node.
4. Multi node. Writes <output_dir>/madengine_task_<job tag>.sh and runs srun bash <task script>. Each task sets NODE_RANK=$SLURM_PROCID, emits the launcher's environment (MAD_MULTI_NODE_RUNNER etc.), rsyncs the project (excluding .git, caches, virtualenvs, old logs) into a node-local workspace madengine_job_<tag>_node_<rank>, builds the same local manifest, sets MIOPEN_USER_DB_PATH=/tmp/.miopen/node_<rank>_rank_<local> and TORCH_ELASTIC_RDZV_TIMEOUT=3600, and calls madengine run with output redirected to madengine-<model>_<tag>_node_<rank>.{out,err}. Only rank 0 collects performance (MAD_COLLECT_METRICS=true); the others set MAD_SKIP_PERF_COLLECTION=true.
5. Artifact collection. On success each node copies its logs, *_env.csv, *.log, multi_results*.csv, results*, gpu_info_*.csv, library_trace.csv, the card's multiple_results file, and the profiler directories rocprof_output, rpd_output, rocm_trace_lite_output, torch_profiler_output, tracelens_output to <output_dir>/<model>/<job tag>/node_<rank>/. On failure only the two logs are copied.
6. Workspace hygiene. A node's workspace is deleted only after a successful run and a complete artifact copy. Workspaces of the same user older than 7 days whose job is no longer in squeue are reclaimed by later jobs.

Environment set by the job script

Node health preflight

deploy() runs SlurmNodeSelector before sbatch when enable_node_check is true, slurm.nodelist is unset, the scheduler is not Spur, and the run is not in place. It lists idle candidate nodes in the partition (honouring exclude and constraint, at most 100 candidates) and probes them one by one with srun --nodelist=<node> --overlap until enough are clean.

• Unhealthy nodes are appended to the job's --exclude. With enough clean nodes the job is pinned to them via --nodelist.
• Multi-node gate: fewer clean nodes than nodes fails the run ("Not enough clean nodes") unless allow_submit_without_clean_nodes is true. Single-node runs are never gated.
• Inconclusive checks stand down: if every examined node was unreachable/unknown the check is disregarded (no gate, no pin); measured dirty nodes stay excluded.
• auto_cleanup_nodes stops only containers labelled madengine.session and kills Ray, vLLM and SGLang processes on a dirty node.
• Probe jobs are named madengine_nodecheck_<pid>_<time> and always cancelled afterwards.

Admins can additionally install scripts/slurm/epilog.sh as a SLURM epilog to clear stray GPU processes after every job (setup notes in scripts/slurm/SLURM_EPILOG_SETUP.md).

Submission, monitoring and results

• Submit: sbatch <output_dir>/madengine_<model>.sh. Transient Raft-leader errors ("not the raft leader", "no leader elected", "service is currently unavailable") are retried up to 6 times, 5 s apart.
• Monitor: BaseDeployment polls every 30 s. squeue -j states RUNNING, PENDING, CONFIGURING, COMPLETING mean keep waiting; COMPLETED is success; any other state (FAILED, CANCELLED, TIMEOUT, NODE_FAIL, ...) is failure. When the job has left the queue the final state comes from sacct -j <id> -n -X -o State, retried 3 times 5 s apart; -X is dropped if the scheduler rejects it. No row at all gives status UNKNOWN, never an assumed success.
• Results are always collected, even for failed jobs, so failed runs are recorded in perf.csv. The login-node process parses the collected per-node logs and multiple_results CSVs under <output_dir>/<model>/<job id>/node_<rank>/, builds one run record and writes it through the same reporting pipeline as local runs (perf.csv, perf_super.csv/json), so a single cumulative file serves local and distributed runs.
• Cancel (Ctrl-C, SIGTERM, CI cancel): the lifecycle handler runs cleanup(), which issues scancel <job id>. Inside an allocation the allocation itself is never cancelled; the in-place process group is signalled instead (see below).
• skip_gpu_arch: for non-Spur SLURM runs the orchestrator drops cards whose skip_gpu_arch names the nodes' architecture before submission (a card on the wrong architecture is reported SKIPPED, not submitted). The filtered manifest is a unique per-run copy <manifest stem>.skip_gpu_arch.<pid>.<uuid>.json beside the build manifest; the build manifest is left untouched, and a failure to write the copy aborts instead of falling back to the original. --disable-skip-gpu-arch turns this off.

Running inside an existing allocation (salloc)

When SLURM_JOB_ID is set madengine detects the allocation and sizes it with scontrol show job (falling back to SLURM_JOB_NUM_NODES, SLURM_NNODES, then the node list). Requesting more nodes than the allocation holds fails; fewer is allowed and prints a note.

• Every in-place run gets a discriminator suffix _<YYYYmmddHHMMSS>_<6 hex> appended to the job tag, so several runs in one allocation never share paths. The suffix appears in node workspaces (madengine_job_<job id><suffix>_node_<rank> under $SLURM_TMPDIR or /tmp), logs, collection directories and the DeepSpeed hostfile /tmp/deepspeed_hostfile_<job id><suffix>.txt.
• The result has skip_monitoring=true (nothing to poll). A non-zero exit is still a failed run, but results are collected so rows from cells that passed are not lost.
• The timeout is resolved like the job's own madengine would (--timeout, then the card, then the default), so a card allowing longer runs is not cut off.
• On timeout or cancel the step's process group gets SIGTERM, then SIGKILL (30 s grace on the normal timeout path; immediately from the signal handler). The allocation is left alone.

Spur scheduler ("scheduler": "spur")

Spur (Crusoe) exposes SLURM-compatible CLI shims but differs in ways that break the stock multi-node flow: srun cannot fan out tasks across nodes (SLURM_PROCID is empty), scontrol show hostname[s] is unsupported, and the Raft-based control plane makes squeue/sacct eventually consistent. SpurDeployment reuses the SLURM template, presets and orchestration, and swaps the fan-out mechanism:

Layered model configuration (mad-config.yaml)

Implemented in deployment/layered_config.py; consumed on the slurm_multi path (_prepare_slurm_multi_script). A card may carry one file that holds what travels with a model: how it is served and what is measured. It resolves to a plain environment dict, so no workload script has to change. Run-level and site-level settings stay in --config groups or cluster.sh; see docs/distributed-config.md.

Kubernetes deployment

Implemented by KubernetesDeployment in deployment/kubernetes.py, composed from mixins. It renders Jinja2 manifests, applies them with the Kubernetes Python client, streams logs, and reads results back through pod logs, kubectl and a results volume. The kubernetes package ships as a core dependency; if it fails to import, the target is unregistered (see Target inference).

Lifecycle (BaseDeployment.execute())

1. validate — cluster reachable, namespace exists, at least one node advertises gpu_resource_name. madengine never creates a namespace. The GPU-node check blocks CPU-only models too (llm-d attach mode skips it).
2. prepare — renders ConfigMap, Job and (multi-node) headless Service.
3. deploy — deletes this job's leftovers, creates the results PVC, Secrets, ConfigMap, Service, then the Job.
4. monitor — polls the Job; with --live-output it streams pod logs. A Job is finished only on the Complete condition (or succeeded >= completions); Failed condition means failure. A non-zero succeeded count alone is not completion.
5. collect results — always runs, even after failure, so failed runs are recorded.
6. cleanup — on deploy failure or Ctrl-C/cancel only: deletes the Job, ConfigMap and Service. After a successful run the Job and its PVC stay unless ttl_seconds_after_finished is set; the next run of the same model replaces them.

Source files

Preset merge order

ConfigLoader.load_k8s_config() deep-merges (later wins; dicts merge, lists and scalars replace, env_vars merge per key):

1. presets/k8s/defaults.json
2. presets/k8s/gpu-vendors/amd.json or nvidia.json, chosen by gpu_vendor (default AMD); sets k8s.gpu_resource_name to amd.com/gpu or nvidia.com/gpu. The NVIDIA preset also sets NCCL env vars.
3. presets/k8s/gpu-vendors/amd-multi-gpu.json — AMD only, when the run is multi-GPU or multi-node: NCCL/RCCL env vars (NCCL_IB_DISABLE=1, NCCL_SOCKET_IFNAME=eth0, GPU_MAX_HW_QUEUES=2, HSA_ENABLE_SDMA=0, …).
4. One profile, chosen from the merged defaults and user config:

   Profile	Selected when	Sets
   multi-node.json	distributed.nnodes > 1	memory 64Gi/128Gi, cpu 16/32, distributed torchrun defaults, NCCL_* env
   multi-gpu.json	gpu_count > 1, or a distributed launcher/enabled, and single node	memory 64Gi/128Gi, cpu 16/32, distributed.launcher=torchrun unless you set one
   single-gpu.json	gpu_count is 1 (the default when unset) and no distributed block	gpu_count=1, memory 16Gi/32Gi, cpu 8/16
   none	e.g. gpu_count: 0 without a distributed block	code fallbacks: memory 128Gi/256Gi, cpu 32/64

5. User configuration (runtime context merged over the manifest's deployment_config).

The loader records which k8s and env_vars keys came only from a preset (_preset_k8s_keys, _preset_env_keys), so self-managed slurm_multi cards are not sized or configured by preset guesses.

k8s sub-keys

Only keys read by the code are listed. Defaults come from presets/k8s/defaults.json unless noted.

Results and data volumes

ReadWriteMany support is detected by reading the StorageClass: a provisioner from a known shared-filesystem family (NFS, CephFS, EFS, Azure Files, Filestore) or an existing ReadWriteMany PersistentVolume of that class. A missing or forbidden class counts as unusable. data_layout: auto works the same for madengine-shared-data: an existing RWX claim is reused, an existing non-RWX claim is left unmounted and each pod gets a local volume. Data providers always mount at /data.

Secrets and image pulls

• With from_local_credentials and a credential.json in the working directory, deploy creates <job>-runtime (mounted at /workspace/credential.json) and, if it holds usable Docker Hub credentials, <job>-registry-pull. Both carry app=madengine. Without the file, no Secret is created and an empty {} credential payload goes into the ConfigMap.
• There is no top-level secrets context key that turns arbitrary values into a Kubernetes Secret. Put credentials in credential.json, or pre-create a Secret and point k8s.secrets at it.
• The pod must be able to pull the image. A local ci-… tag or a placeholder such as <your-image> is not pullable: build with --registry or pass a registry reference with --use-image, or the run fails with a ConfigurationError naming the push error. require_pinned_image in the context makes pods pull by digest.

Names and multi-node

• Job name is madengine-<sanitized model>, capped at 63 characters (Batch Jobs require the job-name label to equal metadata.name); long names are shortened with a hash. The headless Service, pod subdomain and container name share one DNS-label form of it. The ConfigMap is <job>-config.
• nnodes > 1 makes an Indexed Job (completions=parallelism=nnodes) with a headless Service and pod subdomain so <job>-0 resolves. sglang-disagg always gets the Service and needs at least 3 nodes (pod 0 is a dedicated proxy).
• vLLM and SGLang are per-replica launchers: each node reports its own metric and results are aggregated across pods.
• Launcher shape is resolved from launcher.type / launcher.nnodes (older form), then distributed in the run context, then the manifest's deployment_config.distributed, field by field. A runtime distributed block wins over the manifest, and n_gpus: -1 is not used as a process count.
• The model script directory is bundled recursively (text files only) into the ConfigMap; a payload beyond about 1 MiB is rejected before submission.

Leftovers, cancel and results

• Names are deterministic, so a rerun replaces its own Job, Secrets, ConfigMap, Service, collector pod and results PVC. Everything madengine creates is labelled app=madengine; a same-named object without that label stops the deploy instead of being deleted (older unlabelled Secrets and collector pods are accepted). Use another namespace in that case.
• Results land under ./k8s_results/<job>/<pod>/: pod.log, a pvc/ mirror, and log_csv/ files rebuilt from MADENGINE_PERF_CSV_BEGIN/END blocks the pod printed when no volume copy survived.
• The card's timeout (or --timeout) is applied inside the pod with timeout; 0 means unbounded.
• If the Kubernetes API is unreachable during cleanup, delete by hand: kubectl delete job,configmap,service -l app=madengine -n <ns> (scope with --context if you use several clusters).

Examples

# Minimal: 4 GPUs, one pod (presets supply everything else)
madengine run --manifest-file build_manifest.json \
  --additional-context '{"k8s": {"namespace": "ml-team", "gpu_count": 4}}'

# Two nodes x two GPUs (examples/k8s-configs/minimal/torchrun-multi-node-minimal.json)
{
  "gpu_vendor": "AMD", "guest_os": "UBUNTU",
  "k8s": {"gpu_count": 2},
  "distributed": {"launcher": "torchrun", "nnodes": 2, "nproc_per_node": 2}
}

More shapes are in examples/k8s-configs/ (minimal/ and basic/: torchrun, DeepSpeed, Megatron-LM, TorchTitan, Primus, vLLM, SGLang, SGLang disaggregated, data provider). Some set host_ipc, which is not read.

slurm_multi on Kubernetes

A slurm_multi card ships its own SLURM batch script that fans out with srun and starts one docker run per node. On Kubernetes madengine runs that script unchanged: one pod per node (an Indexed Job with a headless Service), the script on pod 0, and stand-ins for srun, docker and scontrol first on PATH. Code: deployment/k8s_slurm_multi.py and deployment/k8s_shim/. Select it by giving a card (or distributed.launcher) the slurm_multi launcher together with a k8s block.

Pod-spec keys (all under k8s)

Shim coverage

Only the forms the MAD launchers use are implemented. Anything else exits with a message naming the unsupported form, so a launcher change that needs more fails loudly instead of running wrongly.

Environment, results and limits

• Environment: the pod gets the card's env_vars plus the run's env_vars, minus keys only a preset supplied (for example the AMD multi-GPU preset's NCCL_IB_DISABLE), DOCKER_IMAGE_NAME, and LOG_PATH defaulting to /results/logs (shared by all pods). Internal MAD_K8S_* variables carry node count, name prefix, run id and the card script.
• Results: pod 0 publishes the file named by the card's multiple_results (found under the script directory or LOG_PATH), else $LOG_PATH/$SLURM_JOB_ID/perf.csv, else ./perf.csv. madengine reads exactly that file from ./k8s_results/<job>/run-<id>/, splits rows into successful and failed by status, and appends them to ./perf.csv. No published CSV is a failed run, never an empty success.
• Names: pod hostnames are <job>-<index> and the card addresses nodes by them, so dots in card names become dashes and long names are shortened with a hash to keep them valid DNS labels.
• Storage: this path always uses the shared <job>-results claim, so multi-node needs a ReadWriteMany class (or a results_access_mode override); results_layout: per_pod is not used.
• Image requirements: the card's image needs bash and python3. The scripts bundle is capped near 900 KiB of base64 so the ConfigMap stays under 1 MiB; narrow include_script_dirs if it is exceeded.
• The image reference is used as resolved; unlike the standard path it is not checked for being node-pullable.

{
  "k8s": {
    "namespace": "ml-team",
    "gpu_count": 8,
    "volumes": [{"host_path": "/data/models", "mount_path": "/models", "read_only": true}],
    "shm_size": "128Gi"
  },
  "distributed": {"launcher": "slurm_multi", "nnodes": 3}
}

slurm_multi launcher on SLURM

{
  "slurm": {"partition": "gpu", "nodes": 3, "gpus_per_node": 8, "time": "02:00:00"},
  "distributed": {"launcher": "slurm_multi"}
}

Build modes

Compute nodes pull the image, so a registry reference is required for multi-node runs. madengine build resolves it in this order:

The generated wrapper

1. Header: --job-name=madengine-<model>, --nodes/--ntasks = resolved node count, --export=ALL, --gpus-per-node (unless skip_gpus_directive), --time, --exclusive (default), and reservation, nodelist, qos, account when set. slurm.nodes is sized from distributed.nnodes when not set explicitly; an explicit slurm.nodes wins with a warning; invalid nnodes is rejected.
2. unset SLURM_EXCLUSIVE (otherwise every srun inside re-parses it as a step request and fails with "Invalid --exclusive specification"), then module load for slurm.modules.
3. Environment exports, lowest to highest precedence: layered config (mad-config.yaml), manifest context.docker_env_vars, the card's env_vars, then additional_context.env_vars minus keys only a preset supplied (so the multi-node preset's NCCL_IB_DISABLE=1, NCCL_SOCKET_IFNAME=eth0, HSA_ENABLE_SDMA=0 do not override the card's own fabric detection). Also: xP/yD from distributed.sglang_disagg (prefill_nodes/decode_nodes) when the script has not set them, and MAD_TOOLS=<names> when tools are configured and rocprofv3 is available (madengine cannot wrap a script it does not control; the script decides). Values are shell-quoted.
4. Node addressing (stock SLURM): MAD_NODE_IPS (rank-ordered, loopback addresses rejected), IPADDRS, SGLANG_NODE_IPS and MAD_NODE_RANK, each only if not already set. Spur builds the equivalent through its rendezvous directory.
5. Image staging for registry images: one srun --ntasks=$SLURM_NNODES --ntasks-per-node=1 step that logs in using MAD_DOCKERHUB_USER/MAD_DOCKERHUB_PASSWORD (or MAD_DOCKER_USER/MAD_DOCKER_TOKEN; only the names are written to the script, values arrive through --export=ALL), skips nodes that already hold the image, and otherwise pulls. Spur pulls locally in each array task. DOCKER_IMAGE_NAME comes from the registry reference recorded by the push, else an already-resolved card value, else the card's docker_image. With require_pinned_image it is pinned to the build digest.
6. Run: cd into the script's directory, delete a stale multiple_results file (unless it is one of madengine's own cumulative reports), then bash <script> <args> with set +e so the exit code is always captured. The script name and args are shell-quoted. A madengine_<model>_<job id>.complete marker (exit_code=, timestamp=) is written to slurm.output_dir, and the wrapper exits with the script's exit code.

Execution paths

Results aggregation

The card's script writes the full perf schema itself, so it is read directly rather than merged with common run info (re-deriving status from performance would turn a legitimate zero-score FAILURE into SUCCESS). The first source found wins:

1. The card's multiple_results, searched in the script's directory, <output_dir>/<job id>, <output_dir> and the current directory. A file older than this submission (30 s clock-skew slack) is ignored as stale.
2. perf*.csv in slurm.results_dir.
3. /shared_inference/$USER/<job id>/perf.csv (or .../model_blog_logs/<job id>/perf.csv), slurm_output/perf_csv/*<job id>*.csv, then ./perf.csv, retried for about 30 s to ride out NFS propagation.

Rows are then copied (or appended without the header) into ./perf.csv so madengine report to-html and the run summary find them. If the source is ./perf.csv nothing is appended, so rows are never duplicated.

Local execution of a self-managed card

If a self-managed launcher reaches ContainerRunner, it runs the script directly on the host (no Docker): bash <script> (or python3 for .py), working directory = the script's directory, model env_vars and context env_vars injected, and only the keys logged, never the values. The script must be a .sh, .slurm or .py file, or a directory containing run.sh.

llm-d deployment

Benchmarks an llm-d stack (vLLM model servers behind a Gateway API Inference Extension InferencePool). The benchmark client is an ordinary single-pod Kubernetes Job, so LlmdDeployment subclasses KubernetesDeployment and reuses ConfigMap bundling, log streaming, result harvesting and the perf.csv path. Selected by an llm_d key (see Target inference); it cannot be combined with slurm. Code: deployment/llm_d.py, deployment/llm_d_stack.py, deployment/presets/llm-d/defaults.json.

{
  "k8s": {"namespace": "llm-d-bench", "gpu_count": 0},
  "llm_d": {
    "endpoint_url": "http://llm-d-inference-gateway.llm-d-bench.svc.cluster.local:80",
    "model": {"name": "Qwen/Qwen3-32B"}
  }
}

llm_d.model.name is required in both modes. It is the model string sent in requests and recorded in results.

llm_d keys

k8s.gpu_count defaults to 0 under llm-d (the client is CPU-only so it does not compete with the stack it measures); an explicit value wins. With gpu_count: 0 and no distributed block no sizing profile applies, so the client pod requests the fallback 128Gi memory and 32 CPUs; set k8s.memory and k8s.cpu explicitly.

Managed flow

1. Model preparation: ensure madengine-shared-data if the URI names it; run the cache-download Job when cache_pvc is set.
2. Values: madengine generates chart values for what it models (model, replicas, tensor parallelism, GPU count and accelerator type, auth secret, gateway class, service port, image pull secrets from k8s.secrets.image_pull_secret_names). When prefill.replicas > 0 it also generates the llm-d endpoint-picker configuration, a pinned disaggregation sidecar and NIXL KV-transfer settings with ROCm UCX transports (rocm_copy,rocm_ipc,tcp,self,sm). Non-portable layout (anti-affinity, ClusterIP gateways) belongs in extra_values.
3. Install three releases in order — <prefix>-infra (Gateway), <prefix>-gaie (InferencePool and endpoint picker), <prefix>-modelservice (prefill/decode servers) — with helm upgrade --install, so a rerun converges. Each release is recorded before helm runs so an interrupted install is still torn down.
4. Readiness (second stage above), then endpoint discovery: the Gateway labelled app.kubernetes.io/instance=<prefix>-infra (or the only Gateway in the namespace); its status.addresses and an HTTP listener are preferred. Several candidate Gateways, or none with an address, is an error that tells you to set endpoint_url.
5. Benchmark Job runs with the MAD_LLM_D_* environment below.
6. Teardown runs on success, on failure, on a failed standup, and on cancel (registered as a signal-handler hook). Releases are uninstalled newest first with --ignore-not-found; a failed uninstall prints the manual helm uninstall command and never masks the benchmark result. teardown: false keeps them.

Client environment contract

Model identity and reporting

• The selected card's --model_repo (in its args) fills llm_d.model.hf_repo and model.name when unset. A different explicit value is a ConfigurationError, so a run cannot serve one model and record another. Cards without --model_repo need an explicit model.name.
• Only the first model matched by --tags is benchmarked (the stack serves that model and image); a warning lists the rest.
• Result rows record deployment_type=llm-d, launcher=vllm (for aggregated and disaggregated serving alike), and serving_mode=aggregated|disaggregated (disaggregated when prefill replicas run). n_gpus, nnodes and gpus_per_node describe the serving pods (read from live pods, falling back to the role config), not the client. The Model column stays the MAD card name.
• When the card sets multiple_results, that CSV is used in preference to performance: log lines.

Managed example

{
  "k8s": {"namespace": "llm-d-bench", "gpu_count": 0, "gpu_resource_name": "amd.com/gpu"},
  "llm_d": {
    "model": {"hf_repo": "Qwen/Qwen3-0.6B", "name": "Qwen/Qwen3-0.6B", "size": "5Gi"},
    "prefill": {"replicas": 0},
    "decode": {"replicas": 1, "tensor_parallel": 1, "gpu_count": 1, "image": "rocm/vllm:latest"},
    "charts": {
      "infra": {"ref": "llm-d-infra/llm-d-infra", "version": "v1.4.0"},
      "gaie": {"version": "v1.5.0"},
      "modelservice": {"ref": "llm-d-modelservice/llm-d-modelservice", "version": "v0.4.16"}
    }
  }
}

From examples/llm-d-configs/05-managed-qwen3-0.6b-amd.json (aggregated serving, a cheap end-to-end smoke test). The llm-d-infra/… aliases need helm repo add first; see the file's comments. Run 02-managed-dry-run.json first to inspect generated values. Other examples: 03 disaggregated with a model cache PVC, 04 aggregated with teardown: false and extra_values.

madengine run --tags dummy_llm_d \
  --additional-context "$(cat examples/llm-d-configs/05-managed-qwen3-0.6b-amd.json)"

Launcher matrix

distributed.launcher accepts exactly the names below. Validation (validate_launcher() in deployment/common.py) is case-insensitive and trims whitespace, and runs at every config boundary: CLI validators, BaseDeployment.__init__ (context, manifest deployment_config, and each card), for SLURM, Spur, Kubernetes and llm-d. A second check, ensure_launcher_allowed(), rejects a launcher on a backend that has no arm for it; today that is only slurm_multi on llm-d.

What is and is not checked

• Rejected: any name not in the table (exit 4), and slurm_multi on llm-d (ensure_launcher_allowed(), when LlmdDeployment is constructed).
• Checked when the job is generated, before submission: sglang-disagg node counts and an explicit prefill/decode split (see the disagg tab under Per-launcher configuration).
• Warning only: slurm.nodes differing from distributed.nnodes (the explicit slurm.nodes wins), and GPU-count sources that disagree (utils/gpu_config.py).
• Not cross-checked: two copies of one setting, such as launcher.type vs distributed.launcher or port vs master_port. Each backend reads them in its own order (see the next section), so set only one.
• llm-d is not a launcher: scale lives in llm_d.prefill / llm_d.decode (see llm-d).

Per-launcher configuration

Common keys under distributed: launcher, nnodes (default: the resolved node count), nproc_per_node (default: resolved GPUs per node), backend (exported as DISTRIBUTED_BACKEND, default nccl), and the rendezvous port as port or master_port (default 29500; port wins on SLURM, master_port on Kubernetes, so set only one). The runtime context overrides the manifest's saved deployment_config, and a card's distributed.launcher is used only when the deployment config names none (resolve_launcher_from_sources).

{
  "slurm": {"partition": "gpu", "nodes": 4, "gpus_per_node": 8, "time": "24:00:00"},
  "distributed": {"launcher": "torchrun", "nnodes": 4, "nproc_per_node": 8, "backend": "nccl", "port": 29500},
  "env_vars": {"NCCL_DEBUG": "WARN", "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1"}
}

{
  "slurm": {"partition": "gpu", "nodes": 8, "gpus_per_node": 8, "time": "48:00:00"},
  "distributed": {"launcher": "megatron-lm", "nnodes": 8, "nproc_per_node": 8}
}

{
  "slurm": {"partition": "gpu", "nodes": 4, "gpus_per_node": 8, "time": "24:00:00"},
  "distributed": {"launcher": "torchtitan", "nnodes": 4, "nproc_per_node": 8}
}

{
  "slurm": {"partition": "gpu", "nodes": 8, "gpus_per_node": 8, "time": "48:00:00", "reservation": "ml-priority"},
  "distributed": {"launcher": "deepspeed", "nnodes": 8, "nproc_per_node": 8, "backend": "nccl"}
}

{
  "slurm": {"partition": "gpu", "nodes": 2, "gpus_per_node": 8, "time": "12:00:00"},
  "distributed": {"launcher": "vllm", "nnodes": 2, "nproc_per_node": 8}
}

{
  "slurm": {"partition": "gpu", "nodes": 2, "gpus_per_node": 8, "time": "06:00:00"},
  "distributed": {"launcher": "sglang", "nnodes": 2, "nproc_per_node": 8}
}

{
  "slurm": {"partition": "gpu", "nodes": 5, "gpus_per_node": 8, "time": "04:00:00"},
  "distributed": {
    "launcher": "sglang-disagg", "nnodes": 5, "nproc_per_node": 8,
    "sglang_disagg": {"prefill_nodes": 2, "decode_nodes": 2}
  }
}

{
  "distributed": {
    "launcher": "primus",
    "primus": {"config_path": "examples/megatron/configs/MI300X/deepseek_v2_lite-BF16-pretrain.yaml"}
  }
}

Config recipes

Working shapes for common scenarios. Partition names, registries and tags are placeholders: replace them with your own. Every key shown is read by the code; more complete files live in examples/slurm-configs/, examples/k8s-configs/, examples/profiling-configs/ and examples/configs/. --additional-context-file and --additional-context can be combined (the inline string is deep-merged over the file); --config cannot be combined with either.

madengine run --tags llama3 \
  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU", "docker_gpus": "0"}'

madengine run --tags megatron-llama3 \
  --additional-context '{
    "gpu_vendor": "AMD",
    "guest_os": "UBUNTU",
    "docker_gpus": "0,1,2,3",
    "distributed": {"launcher": "megatron-lm"}
  }'

cat > slurm-single.json <<'EOF'
{
  "slurm": {"partition": "gpu", "nodes": 1, "gpus_per_node": 8, "time": "12:00:00", "exclusive": true},
  "distributed": {"launcher": "torchrun", "nnodes": 1, "nproc_per_node": 8}
}
EOF
madengine build --tags llama3 --registry registry.example.com/ml
madengine run --manifest-file build_manifest.json \
  --additional-context-file slurm-single.json

cat > slurm-multi.json <<'EOF'
{
  "slurm": {
    "partition": "gpu", "nodes": 4, "gpus_per_node": 8, "time": "24:00:00",
    "exclusive": true, "reservation": "ml-training-q1", "network_interface": "ib0"
  },
  "distributed": {"launcher": "deepspeed", "nnodes": 4, "nproc_per_node": 8, "backend": "nccl"},
  "env_vars": {"NCCL_IB_DISABLE": "0", "NCCL_DEBUG": "WARN", "HSA_ENABLE_SDMA": "0"}
}
EOF
madengine run --manifest-file build_manifest.json \
  --additional-context-file slurm-multi.json

salloc -N 2 --gpus-per-node=8 --exclusive
# inside the allocation shell, on the login node:
madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {"partition": "gpu", "nodes": 2, "gpus_per_node": 8},
    "distributed": {"launcher": "torchrun", "nnodes": 2, "nproc_per_node": 8}
  }'

{
  "slurm": {
    "scheduler": "spur",
    "partition": "gpu", "nodes": 2, "gpus_per_node": 8, "time": "04:00:00",
    "rendezvous_timeout": 1800
  },
  "distributed": {"launcher": "torchrun", "nnodes": 2, "nproc_per_node": 8}
}

madengine run --tags llama3-infer \
  --additional-context '{
    "k8s": {"namespace": "ml-team", "gpu_count": 4, "memory": "128Gi", "cpu": "32"}
  }'

{
  "k8s": {
    "namespace": "ml-team", "gpu_count": 4,
    "memory": "256Gi", "memory_limit": "384Gi", "cpu": "64", "cpu_limit": "96"
  },
  "distributed": {"launcher": "vllm", "nnodes": 2, "nproc_per_node": 4, "master_port": 29500},
  "env_vars": {"PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True"}
}

{
  "slurm": {"partition": "gpu", "nodes": 3, "gpus_per_node": 8, "time": "04:00:00"},
  "distributed": {
    "launcher": "sglang-disagg", "nnodes": 3, "nproc_per_node": 8,
    "sglang_disagg": {"prefill_nodes": 1, "decode_nodes": 1}
  }
}

# Build modes: pick one
madengine build --tags pyt_sglang_disagg --use-image registry.example.com/sglang:latest
madengine build --tags pyt_sglang_disagg --use-image auto          # DOCKER_IMAGE_NAME from the card
madengine build --tags pyt_sglang_disagg --registry registry.example.com/ml --build-on-compute

madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {"partition": "gpu", "nodes": 3, "gpus_per_node": 8, "time": "02:00:00"},
    "distributed": {"launcher": "slurm_multi"}
  }'

madengine run --tags llama3 \
  --additional-context '{
    "gpu_vendor": "AMD",
    "tools": [
      {"name": "rocprofv3_compute"},
      {"name": "gpu_info_power_profiler", "env_vars": {"POWER_DEVICE": "all", "POWER_SAMPLING_RATE": "0.1"}}
    ]
  }'

Profiling & tracing tools

Tools are declared in additional_context.tools as a list of objects and resolved against scripts/common/tools.json (override with madengine run --tools-config FILE). Each definition bundles pre_scripts, a command wrapper (cmd), env_vars and post_scripts. Implementation: execution/container_runner.py::apply_tools(); helper scripts live in scripts/common/{pre_scripts,post_scripts,tools}/.

{
  "tools": [
    {"name": "rocprofv3_compute"},
    {"name": "gpu_info_power_profiler",
     "env_vars": {"POWER_SAMPLING_RATE": "0.5"}},
    {"name": "tracelens", "env_vars": {"TRACELENS_MODE": "rocprof"}}
  ]
}

How tools are applied

Tool catalog

tools.json defines every entry below. All rocprof-family tools except rocprofv3 go through tools/rocprof_wrapper.sh, which uses legacy rocprof on ROCm < 7.0 and prefers rocprofv3 on ≥ 7.0. Always keep the -- separator between profiler options and the command.

GPU kernel / API profilers → rocprof_output/ (rpd_output/ for rpd)

Counter files: scripts/common/tools/counters/{compute_bound,memory_bound,communication_bound,full_profile,instruction_mix}.txt.

Other tracers and monitors

rocm-trace-lite

RTL ships as GitHub release wheels, not on PyPI. The rocm_trace_lite pre-script (pre_scripts/trace.sh) installs the pinned v0.3.7 linux_x86_64 wheel by default and needs outbound HTTPS to github.com unless the package is baked into the image.

torch.profiler on unmodified workloads (dynolog) and TraceLens

To analyze traces after the fact on the host, use madengine report tracelens and tracelens-compare (see report). Example contexts: examples/profiling-configs/ (torch_profiler_tracelens.json, tracelens_rocprofv3.json, rocm_trace_lite*.json, rocprofv3_*.json).

Output collection

rocEnvTool (system environment capture)

Unless disabled, pre_scripts/run_rocenv_tool.sh runs before the model as <model>_env <mode> <GUEST_OS>. It is controlled by the CLI flag --sys-env-details (default on) and the context key gen_sys_env_details (the context wins). If the script is not reachable it is skipped with a message rather than failing the run.

ROCm & GPU utilities

Host-side GPU discovery lives in utils/ (rocm_path_resolver, therock_markers, rocm_tool_manager, nvidia_tool_manager, gpu_tool_factory, gpu_validator, gpu_config, gpu_memory_gate) and is driven by core/context.py. Two independent ROCm roots are resolved: the host's and the container's.

Tool selection and GPU discovery

--wait-for-gpu-memory

utils/gpu_memory_gate.py blocks a local run until every GPU reports at most 512 MB in use. It runs after the image build and before the first container starts, to absorb slow driver release from a previous job.

• Fails closed: if memory cannot be queried (amd-smi metric --mem-usage --json, then rocm-smi --showmeminfo vram --json; nvidia-smi on NVIDIA) or returns no devices, the run is refused with an ExecutionError.
• The error names each busy GPU and its used MB, and hints at a container left up by --keep-alive.
• On SLURM and Kubernetes the flag is ignored with a warning, as are --keep-alive, --keep-model-dir and --skip-model-run.

Environment variables

Three groups: variables read by madengine on the host, variables set by madengine for containers, jobs and pods, and the variables the SLURM job script exports. Context keys such as MAD_ROCM_PATH and MAD_CONTAINER_IMAGE live in additional_context, not in the process environment; see additional_context.

Read by madengine