madengine — Codebase Wiki

What it does

madengine is a Typer-based CLI (madengine) that discovers models from a MAD package, builds Docker images, and runs them either locally or on distributed backends (Kubernetes, SLURM). It writes performance results to perf.csv and can generate HTML reports or upload to MongoDB.

Entry point: src/madengine/cli/app.py::cli_main (registered as the madengine console script in pyproject.toml).

Why this branch matters

The add_slurm_multi_launcher branch adds a self-managed multi-node SLURM launcher so that workloads with their own per-node Docker orchestration (e.g. SGLang Disaggregated prefill + decode + proxy) can run via a thin wrapper SBATCH that does not nest Docker inside the job step. It adds --use-image / --build-on-compute build modes, a registry gate, parallel image pull, and a bash-in-salloc execution path.

Setup

pip install -e ".[dev]"      # base + dev
pip install -e ".[all]"      # + kubernetes
pre-commit install

Test & quality

pytest                            # all tests
pytest tests/unit/test_slurm_multi.py -v
pytest --cov=src/madengine --cov-report=html
pytest -m "not slow"
black src/ tests/ && isort src/ tests/
flake8 src/ tests/
mypy src/madengine
pre-commit run --all-files

Build flow

1. madengine build → BuildOrchestrator.execute()
2. DiscoverModels resolves --tags against the MAD package (root models.json, scripts/{dir}/models.json, or scripts/{dir}/get_models_json.py).
3. Each model is materialised through Context (system + user additional_context) and passed to DockerBuilder.
4. Optionally tags & pushes to --registry.
5. Writes build_manifest.json consumed by run.

Special build modes on this branch:

• --use-image [IMAGE|auto] — skip local build, use a prebuilt image (auto resolves env_vars.DOCKER_IMAGE_NAME from the model card). Mutually exclusive with --registry and --build-on-compute.
• --build-on-compute — build on a SLURM compute node and push to --registry; manifest carries built_on_compute: true.

Run flow

1. madengine run → RunOrchestrator loads existing manifest or triggers a build.
2. Target inference (Convention over Configuration):
   • "k8s"/"kubernetes" in context → KubernetesDeployment
   • "slurm" in context → SlurmDeployment
   • distributed.launcher == "slurm_multi" → slurm_multi path
   • neither → ContainerRunner (local Docker)
3. scripts/common/ is populated from the package (pre_scripts, post_scripts, tools) and cleaned up afterwards.
4. Per-model results parsed via PERFORMANCE_LOG_PATTERN and appended to perf.csv/perf_entry.csv. Failed runs are still recorded with STATUS=FAILURE.

What it is

A minimal-but-additive SLURM launcher for workloads that orchestrate their own per-node Docker containers via srun — for example SGLang Disaggregated (proxy + prefill + decode topologies) or anything that needs to call srun / scontrol from inside the job script.

Generates a wrapper SBATCH that runs the model's .slurm script directly on baremetal (not inside a container), so the workload can spawn its own per-node containers without the outer job step holding a container open.

How to pick it

{
  "slurm": {"partition":"gpu","nodes":3,"gpus_per_node":8,"time":"02:00:00"},
  "distributed": {"launcher": "slurm_multi"}
  // aliases: "slurm-multi"
}

Honors model-card + context slurm fields: partition, nodes, gpus_per_node, time, exclusive, reservation, nodelist.

unit/

Fast, isolated, mocked. ~28 modules including test_slurm_multi.py, test_shell_quoting.py, test_error_handling.py, test_k8s.py, test_rocm_path.py, test_validators.py.

integration/

Real Docker / GPU / platform calls. Includes test_docker_integration.py, test_container_execution.py, test_gpu_management.py, test_orchestrator_workflows.py, test_profiling_tools_config.py.

e2e/

Full workflows: test_build_workflows.py, test_run_workflows.py, test_profiling_workflows.py, test_data_workflows.py, test_execution_features.py, test_scripting_workflows.py.