ROCm stack health

Before you start

• One AMD GPU with working ROCm drivers
• Python 3.10+ and a PyTorch build for your ROCm version
• Network access to github.com/ROCm/aorta (clone + dev wheels)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=1, f'need 1 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/ci/gpu-smoke.yaml --dry-run

4. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

aorta sweep run --recipe recipes/ci/gpu-smoke.yaml --output-dir triage_results/repro/gpu_smoke --strict

5. Read the smoke matrix

RUN_DIR="$(find triage_results/repro/gpu_smoke -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

Success criteria

Rank 0 prints `Wrote matrix to ...`. matrix.md shows every cell completed; mean_step_time_ms is recorded in perf.md. Nightly gates pass/fail against the blessed gpu_smoke::baseline-local baseline; step time is recorded but not performance-thresholded.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• One AMD GPU with enough VRAM for the built-in smoke model
• PyTorch with ROCm (the recipe uses AORTA's RepeatedBlockModel locally)
• Same AORTA wheel channel as nightly when comparing checksums

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=1, f'need 1 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/inference/example-inference-smoke.yaml --dry-run

4. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

aorta sweep run --recipe recipes/inference/example-inference-smoke.yaml --output-dir triage_results/repro/inference_offline --strict

5. Inspect latency, throughput, and checksum artifacts

RUN_DIR="$(find triage_results/repro/inference_offline -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "tokens_per_sec|prefill|decode|checksum" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

Recipe completes with --strict. logits_checksum is the active correctness gate in regression_baselines.yaml; latency and throughput are recorded on the dashboard but not perf-gated yet.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• One gfx950 (MI355X) or gfx1250 GPU — the TokenSpeed image targets these
• A working docker client and daemon: the engine runs in its own container
• A node-local work_dir — an NFS home under root-squash cannot be bind-mounted
• Egress to the Hugging Face Hub, or a pre-populated cache plus hf_offline

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=1, f'need 1 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Serving-specific: pull the pinned engine image

docker pull lightseekorg/tokenspeed-amd@sha256:60c12e37c01496891053b9c30c4204e5d1cf9b4b641859d3aadcbd95bccc7c78

4. Serving-specific: pre-warm the model cache as the running uid

# The path is load-bearing. The workload derives its cache from

# workload_config.hf_home, falling back to <work_dir>/u<uid>/hf --

# it does not read an inherited HF_HOME, so a download into any

# other directory populates a cache the run never mounts and the

# first cell downloads the weights again regardless.

# run_as_current_user defaults to true, so download as the uid

# that will run the sweep: a cache populated by a root container

# leaves the trial failing with PermissionError.

mkdir -p /tmp/ts-work-serve/u$(id -u)/hf

HF_HOME=/tmp/ts-work-serve/u$(id -u)/hf hf download Qwen/Qwen3-0.6B

5. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/tokenspeed/tokenspeed-serve-bench-smoke.yaml --dry-run

6. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

aorta sweep run --recipe recipes/tokenspeed/tokenspeed-serve-bench-smoke.yaml --output-dir triage_results/repro/tokenspeed_serve_smoke --strict

7. Inspect serving latency and throughput artifacts

RUN_DIR="$(find triage_results/repro/tokenspeed_serve_smoke -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "ttft|tpot|output_throughput|completed_total" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

Both cells pass with completed_total == num_prompts * steps and failed_total == 0. median_ttft_ms / median_tpot_ms / output_throughput appear in perf.md. Two metrics are gated on both cells — median_tpot_ms and p99_itl_ms, ceilings from the 2026-09-08..09-17 window — so a run over either fails the nightly, and the breaching observation is still recorded and charted in this cell's history for diagnosis. Every other serving metric is still record-only, including step_time_ms.max; see docs/tokenspeed-gating-rollout.md for which metrics get bounds and why bring-up time never does (189-379s on one node with nothing changed).

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two AMD GPUs on one node, visible to a single PyTorch process group
• RCCL usable between the two devices (check PCIe/xGMI topology)
• torchrun on PATH (ships with PyTorch)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --output-dir triage_results/repro/training_ddp --strict

6. Confirm both ranks finished and step times recorded

RUN_DIR="$(find triage_results/repro/training_ddp -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50|step_time_p99" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Eight AMD GPUs on one node (nightly reference: single MI350 node)
• torchrun --standalone --nproc_per_node=8 must bind one rank per GPU
• Enough host memory for eight concurrent training workers

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --output-dir triage_results/repro/training_ddp_8gpu --strict

6. Check weak-scaling step times on eight ranks

RUN_DIR="$(find triage_results/repro/training_ddp_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet. Compare 2-GPU vs 8-GPU step_time_p50 in the dashboard scaling section.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two AMD GPUs with FSDP-compatible PyTorch build
• Same torchrun/NCCL setup as DDP (FSDP still uses dist.init_process_group)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --output-dir triage_results/repro/training_fsdp --strict

6. Read FSDP step-time artifacts

RUN_DIR="$(find triage_results/repro/training_fsdp -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50|step_time_p99" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Full eight-GPU node (nightly runs on a single MI350 host)
• FSDP requires stable NCCL/RCCL across all eight ranks

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --output-dir triage_results/repro/training_fsdp_8gpu --strict

6. Confirm eight-rank FSDP artifacts

RUN_DIR="$(find triage_results/repro/training_fsdp_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet. Use the dashboard scaling table to compare 2-GPU vs 8-GPU FSDP efficiency.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two GPUs — determinism is checked across ranks in one job
• Identical random seeds and deterministic PyTorch ops where required

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --output-dir triage_results/repro/llm_determinism --strict

6. Confirm ranks_with_divergence from perf.md or matrix.json

RUN_DIR="$(find triage_results/repro/llm_determinism -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "ranks_with_divergence|diverge" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'ranks_with_divergence' in metrics:
            values.append(metrics['ranks_with_divergence'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

ranks_with_divergence must be 0 for a pass. Any non-zero value means silent numerical corruption between ranks and should fail nightly CI.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Eight-GPU node — catches determinism bugs that only appear at scale
• Stable RCCL collectives; any rank mismatch fails the workload

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --output-dir triage_results/repro/llm_determinism_8gpu --strict

6. Confirm ranks_with_divergence is zero on all eight ranks

RUN_DIR="$(find triage_results/repro/llm_determinism_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "ranks_with_divergence|diverge" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'ranks_with_divergence' in metrics:
            values.append(metrics['ranks_with_divergence'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

ranks_with_divergence == 0 on every cell. This is a correctness gate, not a performance benchmark.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two GPUs — race workload stresses concurrent RCCL + layer checksums
• Race workloads use fresh process isolation per trial (see recipe)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Race-specific: reserve trial master ports on static launchers

export AORTA_TRIAL_MASTER_PORT_BASE=30000

# Required for srun / static torchrun without elastic agent store

5. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/race/race_smoke.yaml --dry-run

6. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/race/race_smoke.yaml --output-dir triage_results/repro/race --strict

7. Check layer checksum mismatches in matrix output

RUN_DIR="$(find triage_results/repro/race -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "layer_checksum_mismatch" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'layer_checksum_mismatches' in metrics:
            values.append(metrics['layer_checksum_mismatches'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

layer_checksum_mismatches must be 0. Non-zero values indicate detected races or silent corruption in distributed layers.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Eight-GPU node — amplifies timing races vs the 2-GPU smoke
• Export AORTA_TRIAL_MASTER_PORT_BASE on static launchers (see README-running-recipes)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout e45a9ae778dfbb0796da23ad516befc1939f7266

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260929' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Race-specific: reserve trial master ports on static launchers

export AORTA_TRIAL_MASTER_PORT_BASE=30000

5. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/race/race_smoke.yaml --dry-run

6. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/race/race_smoke.yaml --output-dir triage_results/repro/race_8gpu --strict

7. Confirm zero checksum mismatches at 8-GPU scale

RUN_DIR="$(find triage_results/repro/race_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "layer_checksum_mismatch" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'layer_checksum_mismatches' in metrics:
            values.append(metrics['layer_checksum_mismatches'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

layer_checksum_mismatches == 0 for every cell. Any failure warrants inspecting per-trial JSON under cells/.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.