madengine — Codebase Wiki

Overview

Quick start

# 1. Install
pip install -e ".[dev]"

# 2. Discover available models
madengine discover --tags dummy

# 3. Build + run (single command)
madengine run --tags dummy \
  --additional-context '{"gpu_vendor":"AMD","guest_os":"UBUNTU"}'

# 4. Build only, then run from manifest
madengine build --tags llama3 --registry registry.example.com/ml
madengine run --manifest-file build_manifest.json \
  --additional-context '{"docker_gpus":"0,1,2,3"}'

# Single-node K8s (minimal — defaults applied from presets/k8s/)
madengine run --tags llama3 \
  --additional-context '{"k8s":{"gpu_count":4}}'

# Multi-node vLLM on K8s
madengine run --tags vllm-serve \
  --additional-context '{
    "k8s": {"namespace":"ml-team","gpu_count":8},
    "distributed": {"launcher":"vllm","nnodes":2,"nproc_per_node":4}
  }'

# K8s with NFS data PVC and secrets
madengine run --tags model \
  --additional-context '{
    "k8s": {"namespace":"ml","gpu_count":8,"data_storage_class":"nfs-banff"},
    "secrets": {"HF_TOKEN":"hf_xxx","WANDB_API_KEY":"yyy"}
  }'

# Single-node SLURM (build on login node, deploy via sbatch)
madengine build --tags llama3 --registry registry.example.com/ml
madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {"partition":"gpu","nodes":1,"gpus_per_node":8,"time":"12:00:00"}
  }'

# Multi-node torchrun
madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {"partition":"gpu","nodes":4,"gpus_per_node":8,"time":"24:00:00"},
    "distributed": {"launcher":"torchrun","nnodes":4,"nproc_per_node":8}
  }'

# DeepSpeed with reservation
madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {"partition":"gpu","nodes":8,"gpus_per_node":8,
              "time":"48:00:00","reservation":"ml-training"},
    "distributed": {"launcher":"deepspeed","nnodes":8,"nproc_per_node":8}
  }'

# SGLang Disaggregated (3+ nodes: proxy + prefill + decode)
madengine run --tags pyt_sglang_disagg_qwen3-32b \
  --additional-context '{
    "slurm": {"partition":"gpu","nodes":3,"gpus_per_node":8,"time":"02:00:00"},
    "distributed": {"launcher":"slurm_multi"}
  }'

# Build options for slurm_multi models:
# Option A — use pre-built registry image (skip local build)
madengine build --tags pyt_sglang_disagg --use-image registry.io/sglang:latest

# Option B — auto-resolve DOCKER_IMAGE_NAME from model card
madengine build --tags pyt_sglang_disagg --use-image auto

# Option C — build on compute node, push, then run pulls in parallel
madengine build --tags pyt_sglang_disagg \
  --registry registry.io/ml --build-on-compute

# Store configuration in a JSON file and reference it
cat > my_run.json <<'EOF'
{
  "gpu_vendor": "AMD",
  "guest_os": "UBUNTU",
  "slurm": {
    "partition": "gpu",
    "nodes": 4,
    "gpus_per_node": 8,
    "time": "24:00:00",
    "exclusive": true
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 4,
    "nproc_per_node": 8,
    "backend": "nccl"
  },
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "HSA_ENABLE_SDMA": "0"
  },
  "tools": [{"name": "rocprofv3_compute"}]
}
EOF

madengine run --tags llama3 --additional-context-file my_run.json

Install & dev

# Base install (includes dev tools)
pip install -e ".[dev]"

# With Kubernetes support
pip install -e ".[all]"

# Enable pre-commit hooks
pre-commit install

pytest                           # all tests
pytest tests/unit/ -v            # unit only
pytest tests/unit/test_slurm_multi.py -v
pytest --cov=src/madengine --cov-report=html
pytest -m "not slow"             # skip slow tests
pytest -m "unit and amd"         # combined markers

black src/ tests/
isort src/ tests/
flake8 src/ tests/
mypy src/madengine
pre-commit run --all-files

5-layer architecture

Each layer talks only to the layers below it. Layers are color-coded throughout this wiki.

Architecture diagram

Key data flows

CLI — discover

Lists and validates model definitions without building or running.

madengine discover [OPTIONS]

  --tags TEXT              Comma-separated tags/names to filter  [required]
  --verbose / --no-verbose Show full model JSON  [default: no-verbose]

Tag syntax

Discovery sources (checked in order per directory)

1. Root models.json
2. scripts/{dir}/models.json (static list)
3. scripts/{dir}/get_models_json.py — dynamic; must export list_models() → List[CustomModel]

CLI — build

Builds Docker images for discovered models and writes build_manifest.json.

madengine build [OPTIONS]

  --tags TEXT                    Tags to select models (mutually exclusive with --batch-manifest)
  --batch-manifest FILE          JSON file of multiple tag groups to build in sequence
  --registry TEXT                Push built images to this registry URL
  --target-archs TEXT            Comma-separated GPU arch list (e.g. "gfx90a,gfx942")
  --use-image [IMAGE|auto]       Skip local build; use named image or auto-resolve from model card
  --build-on-compute             Build on SLURM compute node + push (requires --registry)
  --additional-context TEXT      Python dict / JSON string of context overrides
  --additional-context-file FILE Path to a JSON context file (mutually exclusive with --additional-context)
  --clean-docker-cache           Pass --no-cache to docker build
  --manifest-output FILE         Output path for build_manifest.json  [default: build_manifest.json]
  --summary-output FILE          Output path for build summary JSON
  --live-output / --no-live-output   Stream docker build output line by line  [default: no-live-output]
  --verbose / --no-verbose

--use-image modes

CLI — run

Runs models from a manifest (build if needed) and writes perf.csv.

madengine run [OPTIONS]

  --tags TEXT                    Select models (triggers build if no manifest)
  --manifest-file FILE           Use existing manifest; skip build  [default: build_manifest.json]
  --registry TEXT                Registry for image pull auth
  --timeout INT                  Seconds per model; -1=7200s default, 0=disabled
  --additional-context TEXT      Python dict or JSON string
  --additional-context-file FILE JSON file (mutually exclusive with --additional-context)
  --keep-alive                   Leave container running after model completes
  --keep-model-dir               Do not clean up model directory copy
  --clean-docker-cache           Remove docker image before pull (SLURM mode)
  --skip-model-run               Build/pull only; skip execution
  --manifest-output FILE
  --summary-output FILE
  --live-output / --no-live-output  Stream container output  [default: no-live-output]
  --output FILE                  Redirect container stdout to file
  --tools-json-file-name FILE    Tools config  [default: ./scripts/common/tools.json]
  --generate-sys-env-details / --no-generate-sys-env-details
  --force-mirror-local           Force ContainerRunner even in SLURM/K8s context
  --disable-skip-gpu-arch        Ignore skip_gpu_arch model field
  --cleanup-perf                 Remove existing perf.csv before run
  --verbose / --no-verbose

Timeout resolution

CLI — report & database

# Convert perf.csv to HTML
madengine report to-html --csv-file perf.csv

# Generate consolidated email report
madengine report to-email \
  --directory ./results \
  --output run_results.html

madengine database \
  --csv-file perf.csv \
  --database-name benchmarks \
  --collection-name runs

Exit codes CI contract

Defined in src/madengine/cli/constants.py::ExitCode. Use these in CI pipelines instead of log scraping.

additional_context — configuration spine

--additional-context accepts a Python dict string (parsed with ast.literal_eval, not json.loads) or --additional-context-file accepts a JSON file. The dict is deep-merged into Context.ctx alongside system-detected values.

SLURM sub-keys (slurm dict)

Kubernetes sub-keys (k8s dict)

Model definition — models.json

Each model definition lives in a models.json file (or is returned by get_models_json.py::list_models()). Fields map to the CustomModel dataclass in utils/discover_models.py.

{
  "name": "llama3-8b-train",          // Unique model identifier
  "dockerfile": "docker/Dockerfile.ubuntu.amd",
  "dockercontext": ".",               // Build context dir (relative to scripts dir)
  "scripts": "scripts/llama3/train.sh",
  "url": "https://github.com/org/repo",
  "cred": "hf_token",                 // Credential key from credential.json
  "owner": "ml-team",
  "data": "llama3-dataset",           // Data identifier for DataProvider
  "n_gpus": "8",                      // "-1" = all available; "0" = CPU-only
  "timeout": 14400,                   // Seconds; overridden by --timeout CLI flag
  "training_precision": "bf16",
  "tags": ["llama3", "training", "amd"],
  "args": "--batch-size 4 --seq-len 4096",
  "multiple_results": "results.csv",  // CSV file with multiple perf rows
  "skip_gpu_arch": "gfx908,gfx1100", // Comma-list of archs to skip this model on
  "additional_docker_run_options": "--shm-size 64g",
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 2,
    "nproc_per_node": 8
  },
  "env_vars": {
    "HF_TOKEN": "auto",              // Injected into container env
    "DOCKER_IMAGE_NAME": "reg/img"   // Used by slurm_multi parallel pull
  }
}

Key field notes

Build manifest — build_manifest.json

Written by madengine build, consumed by madengine run. Pass with --manifest-file.

{
  "built_images": {
    "ci-llama3_Dockerfile.ubuntu.amd": {
      "docker_image": "registry.io/ml/ci-llama3:sha256-abc",
      "docker_sha":   "sha256:abc123",
      "build_duration": 183.4
    }
  },
  "built_models": {
    "ci-llama3_Dockerfile.ubuntu.amd": {
      "name":          "llama3-8b-train",
      "dockerfile":    "docker/Dockerfile.ubuntu.amd",
      "docker_image":  "ci-llama3_Dockerfile.ubuntu.amd",
      "docker_sha":    "sha256:abc123",
      "build_duration": 183.4,
      "scripts":       "scripts/llama3/train.sh",
      "args":          "--batch-size 4",
      "tags":          ["llama3","training"],
      "n_gpus":        "8",
      "timeout":       14400,
      "skip_gpu_arch": "",
      "multiple_results": "",
      "distributed":   {"launcher":"torchrun","nnodes":2,"nproc_per_node":8},
      "env_vars":      {"DOCKER_IMAGE_NAME":"registry.io/ml/ci-llama3:sha256-abc"},
      "built_on_compute": false
    }
  },
  "context": {
    "gpu_vendor": "AMD",
    "guest_os":   "UBUNTU",
    "docker_env_vars": {"MAD_GPU_VENDOR":"AMD","MAD_SYSTEM_NGPUS":"8"},
    "docker_build_arg": {}
  },
  "deployment_config": {
    "target":  "slurm",
    "slurm":   {"partition":"gpu","nodes":4,"gpus_per_node":8,"time":"24:00:00"},
    "distributed": {"launcher":"torchrun","nnodes":4,"nproc_per_node":8},
    "env_vars": {"NCCL_DEBUG":"WARN"},
    "debug": false
  },
  "summary": {"total":1,"success":1,"failed":0}
}

Deployment target inference

No explicit deploy field needed. RunOrchestrator._infer_deployment_target() inspects the merged context:

Within SLURM deployment, if distributed.launcher == "slurm_multi" (or "slurm-multi"), SlurmDeployment.prepare() takes the slurm_multi path instead of generating the standard Jinja2 template.

SLURM deployment

Implemented in src/madengine/deployment/slurm.py. Generates an sbatch script from a Jinja2 template at src/madengine/deployment/templates/slurm/job.sh.j2.

Preset merge order

ConfigLoader.load_slurm_config() applies three layers (last wins):

1. presets/slurm/defaults.json — base defaults for all SLURM runs
2. presets/slurm/profiles/single-node.json or multi-node.json — profile selected by nodes count
3. User-supplied slurm / distributed / env_vars keys

{
  "gpu_vendor": "AMD",
  "guest_os": "UBUNTU",
  "debug": false,
  "slurm": {
    "partition": "amd-rccl",
    "nodes": 1,
    "gpus_per_node": 8,
    "time": "24:00:00",
    "exclusive": true,
    "modules": []
  },
  "distributed": {
    "backend": "nccl",
    "port": 29500
  },
  "env_vars": {
    "OMP_NUM_THREADS": "8",
    "MIOPEN_FIND_MODE": "1",
    "MIOPEN_USER_DB_PATH": "/tmp/.miopen"
  }
}

{
  "slurm": {"nodes": 2, "gpus_per_node": 8, "time": "24:00:00"},
  "distributed": {"backend": "nccl", "port": 29500},
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "NCCL_DEBUG_SUBSYS": "INIT",
    "NCCL_IB_DISABLE": "0",
    "NCCL_SOCKET_IFNAME": "ib0",
    "TORCH_NCCL_HIGH_PRIORITY": "1",
    "GPU_MAX_HW_QUEUES": "8",
    "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1",
    "NCCL_TIMEOUT": "1200",
    "HSA_ENABLE_SDMA": "0",
    "HSA_FORCE_FINE_GRAIN_PCIE": "1",
    "RCCL_ENABLE_HIPGRAPH": "0"
  }
}

What the SLURM job script does

• Sets MASTER_ADDR via scontrol show hostnames, MASTER_PORT, WORLD_SIZE, NNODES
• Sets per-node HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES / CUDA_VISIBLE_DEVICES (vLLM/SGLang: only HIP_VISIBLE_DEVICES)
• Sets MIOPEN_USER_DB_PATH per-process: /tmp/.miopen/node_${SLURM_PROCID}_rank_${LOCAL_RANK:-0}
• Sets TORCH_ELASTIC_RDZV_TIMEOUT=3600 for PyTorch elastic
• Sets MAD_DEPLOYMENT_TYPE=slurm, MAD_SLURM_JOB_ID, MAD_NODE_RANK, MAD_IN_SLURM_JOB=1
• Multi-node: generates per-node task script; runs via srun bash $TASK_SCRIPT
• Single-node: creates synthetic manifest with deployment_config.target="docker" and calls madengine run

Node health preflight

SlurmNodeSelector runs a health-check srun before the main job unless slurm.nodelist is set (then skipped). Supports slurm.reservation forwarded to srun commands.

Monitoring

Polls squeue every 30 seconds. Terminal states: COMPLETED, FAILED, CANCELLED — a scancel'd job will not loop forever.

Kubernetes deployment

Implemented in src/madengine/deployment/kubernetes.py and 6 focused mixin modules (refactored in v2.0.3). Requires pip install -e ".[kubernetes]".

Mixin modules

Preset merge order

ConfigLoader.load_k8s_config() applies five layers (last wins):

1. presets/k8s/defaults.json — base defaults
2. presets/k8s/gpu-vendors/amd.json or nvidia.json — GPU resource name
3. presets/k8s/gpu-vendors/amd-multi-gpu.json — AMD multi-GPU NCCL env vars (only if AMD + multi-GPU)
4. presets/k8s/profiles/single-gpu.json, multi-gpu.json, or multi-node.json
5. User config

{
  "k8s": {
    "kubeconfig": "~/.kube/config",
    "namespace": "default",
    "image_pull_policy": "Always",
    "backoff_limit": 3,
    "ttl_seconds_after_finished": null,
    "nfs_storage_class": "nfs-banff",
    "storage_class": "nfs-banff",
    "data_storage_class": "nfs-banff",
    "recreate_shared_data_pvc": false,
    "secrets": {
      "strategy": "from_local_credentials",
      "image_pull_secret_names": [],
      "runtime_secret_name": null
    }
  },
  "env_vars": {"OMP_NUM_THREADS": "8"}
}

{
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "NCCL_IB_DISABLE": "0",
    "NCCL_SOCKET_IFNAME": "ib0",
    "TORCH_NCCL_HIGH_PRIORITY": "1",
    "GPU_MAX_HW_QUEUES": "8",
    "HSA_ENABLE_SDMA": "0",
    "MIOPEN_FIND_MODE": "1",
    "MIOPEN_USER_DB_PATH": "/tmp/.miopen",
    "HSA_FORCE_FINE_GRAIN_PCIE": "1",
    "RCCL_ENABLE_HIPGRAPH": "0"
  }
}

Secrets management

# Pass secrets via additional_context
madengine run --tags llm-serve \
  --additional-context '{
    "k8s": {"namespace":"ml","gpu_count":8},
    "secrets": {"HF_TOKEN":"hf_xxx","WANDB_API_KEY":"yyy","S3_KEY":"zzz"}
  }'

Secrets in additional_context.secrets are auto-converted to a K8s Secret object and mounted as environment variables in the job pod. They are never written to perf.csv or build logs.

slurm_multi launcher merged in v2.1.0

{
  "slurm": {
    "partition": "gpu",
    "nodes": 3,
    "gpus_per_node": 8,
    "time": "02:00:00"
  },
  "distributed": {
    "launcher": "slurm_multi"
  }
}

Build modes

Execution paths

• sbatch (default): wrapper SBATCH submitted to SLURM. Head node calls srun docker pull on all nodes in parallel, then runs the model's script.
• bash-in-salloc: if SLURM_JOB_ID env var is set (inside existing salloc), the launcher runs the wrapper synchronously with bash. Sets DeploymentResult.skip_monitoring=True so the monitor poll is skipped.

Results aggregation

_collect_slurm_multi_results() reads per-job CSV from /shared_inference/$USER/$JOBID/perf.csv and writes those rows into cwd/perf.csv (copy if absent, append data rows if present). This ensures display_performance_table and madengine report to-html find results without extra arguments.

Local self-managed execution

When slurm_multi is detected in a non-SLURM context (e.g. local Docker mode), ContainerRunner._run_self_managed() runs the model's script directly on the host. Env vars from model card and additional_context are injected; keys are logged without values to avoid leaking credentials.

Docker --build-context tools= v2.1.0

# In any model Dockerfile
COPY --from=tools rocm_smi/*.py /opt/mad/tools/rocm_smi/
COPY --from=tools gpu_info/*.py /opt/mad/tools/

Launcher matrix

Per-launcher configuration

{
  "slurm": {"partition":"gpu","nodes":4,"gpus_per_node":8,"time":"24:00:00"},
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 4,
    "nproc_per_node": 8,
    "backend": "nccl",
    "port": 29500
  },
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "HSA_ENABLE_SDMA": "0",
    "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1"
  }
}

{
  "slurm": {"partition":"gpu","nodes":8,"gpus_per_node":8,"time":"48:00:00"},
  "distributed": {
    "launcher": "megatron",
    "nnodes": 8,
    "nproc_per_node": 8
  },
  "env_vars": {
    "TENSOR_MODEL_PARALLEL_SIZE": "4",
    "PIPELINE_MODEL_PARALLEL_SIZE": "2",
    "CONTEXT_PARALLEL_SIZE": "1",
    "NCCL_IB_DISABLE": "0"
  }
}

{
  "slurm": {"partition":"gpu","nodes":4,"gpus_per_node":8,"time":"24:00:00"},
  "distributed": {
    "launcher": "torchtitan",
    "nnodes": 4,
    "nproc_per_node": 8
  },
  "env_vars": {
    "TORCHTITAN_TENSOR_PARALLEL_SIZE": "2",
    "TORCHTITAN_FSDP_ENABLED": "true"
  }
}

{
  "slurm": {
    "partition": "gpu",
    "nodes": 8,
    "gpus_per_node": 8,
    "time": "48:00:00",
    "reservation": "ml-priority"
  },
  "distributed": {
    "launcher": "deepspeed",
    "nnodes": 8,
    "nproc_per_node": 8,
    "backend": "nccl"
  },
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "HSA_ENABLE_SDMA": "0"
  }
}

{
  "slurm": {"partition":"gpu","nodes":2,"gpus_per_node":8,"time":"12:00:00"},
  "distributed": {
    "launcher": "vllm",
    "nnodes": 2,
    "nproc_per_node": 8
  },
  "env_vars": {
    "VLLM_TENSOR_PARALLEL_SIZE": "8",
    "VLLM_PIPELINE_PARALLEL_SIZE": "2"
  }
}

{
  "slurm": {"partition":"gpu","nodes":2,"gpus_per_node":8,"time":"06:00:00"},
  "distributed": {
    "launcher": "sglang",
    "nnodes": 2,
    "nproc_per_node": 8
  },
  "env_vars": {
    "SGLANG_TENSOR_PARALLEL_SIZE": "8"
  }
}

{
  "slurm": {
    "partition": "gpu",
    "nodes": 5,
    "gpus_per_node": 8,
    "time": "04:00:00"
  },
  "distributed": {
    "launcher": "sglang_disagg",
    "nnodes": 5,
    "nproc_per_node": 8,
    "sglang_disagg": {
      "prefill_nodes": 2,
      "decode_nodes": 2
    }
  },
  "env_vars": {
    "SGLANG_TP_SIZE": "8"
  }
}

Config recipes

Complete working configurations for common scenarios.

madengine run --tags llama3 \
  --additional-context '{
    "gpu_vendor": "AMD",
    "guest_os": "UBUNTU",
    "docker_gpus": "0"
  }'

madengine run --tags megatron-llama3 \
  --additional-context '{
    "gpu_vendor": "AMD",
    "guest_os": "UBUNTU",
    "docker_env_vars": {
      "TENSOR_MODEL_PARALLEL_SIZE": "4",
      "PIPELINE_MODEL_PARALLEL_SIZE": "2"
    }
  }'

cat > slurm-single.json <<'EOF'
{
  "slurm": {
    "partition": "amd-gpu",
    "nodes": 1,
    "gpus_per_node": 8,
    "time": "12:00:00",
    "exclusive": true
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 1,
    "nproc_per_node": 8
  }
}
EOF
madengine build --tags llama3 --registry registry.example.com/ml
madengine run --manifest-file build_manifest.json \
  --additional-context-file slurm-single.json

cat > slurm-multi.json <<'EOF'
{
  "slurm": {
    "partition": "amd-gpu",
    "nodes": 4,
    "gpus_per_node": 8,
    "time": "24:00:00",
    "exclusive": true,
    "reservation": "ml-training-q1",
    "network_interface": "ib0"
  },
  "distributed": {
    "launcher": "deepspeed",
    "nnodes": 4,
    "nproc_per_node": 8,
    "backend": "nccl"
  },
  "env_vars": {
    "NCCL_IB_DISABLE": "0",
    "NCCL_SOCKET_IFNAME": "ib0",
    "NCCL_DEBUG": "WARN",
    "HSA_ENABLE_SDMA": "0"
  }
}
EOF
madengine run --manifest-file build_manifest.json \
  --additional-context-file slurm-multi.json

madengine run --tags llama3-infer \
  --additional-context '{
    "k8s": {
      "namespace": "ml-team",
      "gpu_count": 4
    }
  }'

madengine run --tags vllm-llama3-70b \
  --additional-context '{
    "k8s": {
      "namespace": "ml-team",
      "gpu_count": 8,
      "host_ipc": true,
      "data_storage_class": "nfs-banff"
    },
    "distributed": {
      "launcher": "vllm",
      "nnodes": 2,
      "nproc_per_node": 8
    },
    "secrets": {"HF_TOKEN": "hf_xxxxxxx"},
    "env_vars": {
      "VLLM_TENSOR_PARALLEL_SIZE": "8",
      "VLLM_PIPELINE_PARALLEL_SIZE": "2"
    }
  }'

madengine build --tags pyt_sglang_disagg --use-image registry.io/sglang:v0.4

madengine run --manifest-file build_manifest.json \
  --additional-context '{
    "slurm": {
      "partition": "amd-gpu",
      "nodes": 3,
      "gpus_per_node": 8,
      "time": "04:00:00"
    },
    "distributed": {
      "launcher": "slurm_multi"
    }
  }'

madengine run --tags llama3 \
  --additional-context '{
    "gpu_vendor": "AMD",
    "tools": [
      {"name": "rocprofv3_compute"}
    ],
    "rocenv_mode": "full"
  }'

  "tools": [
    {"name": "rocprofv3_compute"},
    {"name": "rccl_trace"},
    {"name": "gpu_info_power_profiler"}
  ]

Profiling & tracing tools

Enable via --additional-context '{"tools":[{"name":"…"}]}'. Tools are stackable — list multiple objects. Implemented in scripts/common/tools/ and execution/container_runner.py::apply_tools().

rocm_trace_lite wheel control

rocEnvTool modes

ROCm path resolution

Implemented in src/madengine/utils/rocm_path_resolver.py and src/madengine/core/context.py. Two independent resolution chains run in parallel.

Environment variables

Read by madengine at runtime

Set by madengine in Docker containers

Set by SLURM job script (job.sh.j2)

Error types

Defined in src/madengine/core/errors.py. All inherit from MADEngineError(Exception) which carries: message, category, context (ErrorContext dataclass), cause, recoverable, suggestions (list). Rich panels are used for display.

Module reference

Test layout

Pytest config lives solely in [tool.pytest.ini_options] in pyproject.toml (minversion=7.0).

Contributing & code style

Changelog