ROCm stack health

Before you start

• One AMD GPU with working ROCm drivers
• Python 3.10+ and a PyTorch build for your ROCm version
• Network access to github.com/ROCm/aorta (clone + dev wheels)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=1, f'need 1 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/ci/gpu-smoke.yaml --dry-run

4. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

aorta sweep run --recipe recipes/ci/gpu-smoke.yaml --output-dir triage_results/repro/gpu_smoke --strict

5. Read the smoke matrix

RUN_DIR="$(find triage_results/repro/gpu_smoke -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

Success criteria

Rank 0 prints `Wrote matrix to ...`. matrix.md shows every cell completed; mean_step_time_ms is recorded in perf.md. Nightly gates pass/fail against the blessed gpu_smoke::baseline-local baseline; step time is recorded but not performance-thresholded.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• One AMD GPU with enough VRAM for the built-in smoke model
• PyTorch with ROCm (the recipe uses AORTA's RepeatedBlockModel locally)
• Same AORTA wheel channel as nightly when comparing checksums

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=1, f'need 1 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/inference/example-inference-smoke.yaml --dry-run

4. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

aorta sweep run --recipe recipes/inference/example-inference-smoke.yaml --output-dir triage_results/repro/inference_offline --strict

5. Inspect latency, throughput, and checksum artifacts

RUN_DIR="$(find triage_results/repro/inference_offline -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "tokens_per_sec|prefill|decode|checksum" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

Recipe completes with --strict. logits_checksum is the active correctness gate in regression_baselines.yaml; latency and throughput are recorded on the dashboard but not perf-gated yet.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two AMD GPUs on one node, visible to a single PyTorch process group
• RCCL usable between the two devices (check PCIe/xGMI topology)
• torchrun on PATH (ships with PyTorch)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --output-dir triage_results/repro/training_ddp --strict

6. Confirm both ranks finished and step times recorded

RUN_DIR="$(find triage_results/repro/training_ddp -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50|step_time_p99" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Eight AMD GPUs on one node (nightly reference: single MI350 node)
• torchrun --standalone --nproc_per_node=8 must bind one rank per GPU
• Enough host memory for eight concurrent training workers

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/training/example-training-ddp-smoke.yaml --output-dir triage_results/repro/training_ddp_8gpu --strict

6. Check weak-scaling step times on eight ranks

RUN_DIR="$(find triage_results/repro/training_ddp_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet. Compare 2-GPU vs 8-GPU step_time_p50 in the dashboard scaling section.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two AMD GPUs with FSDP-compatible PyTorch build
• Same torchrun/NCCL setup as DDP (FSDP still uses dist.init_process_group)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --output-dir triage_results/repro/training_fsdp --strict

6. Read FSDP step-time artifacts

RUN_DIR="$(find triage_results/repro/training_fsdp -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50|step_time_p99" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Full eight-GPU node (nightly runs on a single MI350 host)
• FSDP requires stable NCCL/RCCL across all eight ranks

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/training/example-training-fsdp-smoke.yaml --output-dir triage_results/repro/training_fsdp_8gpu --strict

6. Confirm eight-rank FSDP artifacts

RUN_DIR="$(find triage_results/repro/training_fsdp_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 10 -E "step_time_p50" "$RUN_DIR/perf.md" 2>/dev/null || python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d.get(\"cells\",[]), indent=2)[:2000])" "$RUN_DIR/matrix.json"

Success criteria

All ranks complete without error. step_time_p50/p99 appear in perf.md. Nightly currently records training cells (passed/recording only); step-time thresholds are not perf-gated in regression_baselines.yaml yet. Use the dashboard scaling table to compare 2-GPU vs 8-GPU FSDP efficiency.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two GPUs — determinism is checked across ranks in one job
• Identical random seeds and deterministic PyTorch ops where required

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --output-dir triage_results/repro/llm_determinism --strict

6. Confirm ranks_with_divergence from perf.md or matrix.json

RUN_DIR="$(find triage_results/repro/llm_determinism -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "ranks_with_divergence|diverge" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'ranks_with_divergence' in metrics:
            values.append(metrics['ranks_with_divergence'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

ranks_with_divergence must be 0 for a pass. Any non-zero value means silent numerical corruption between ranks and should fail nightly CI.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Eight-GPU node — catches determinism bugs that only appear at scale
• Stable RCCL collectives; any rank mismatch fails the workload

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --dry-run

5. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/llm-determinism/example-llm-determinism.yaml --output-dir triage_results/repro/llm_determinism_8gpu --strict

6. Confirm ranks_with_divergence is zero on all eight ranks

RUN_DIR="$(find triage_results/repro/llm_determinism_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "ranks_with_divergence|diverge" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'ranks_with_divergence' in metrics:
            values.append(metrics['ranks_with_divergence'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

ranks_with_divergence == 0 on every cell. This is a correctness gate, not a performance benchmark.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Two GPUs — race workload stresses concurrent RCCL + layer checksums
• Race workloads use fresh process isolation per trial (see recipe)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=2, f'need 2 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Race-specific: reserve trial master ports on static launchers

export AORTA_TRIAL_MASTER_PORT_BASE=30000

# Required for srun / static torchrun without elastic agent store

5. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/race/race_smoke.yaml --dry-run

6. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=2 $(which aorta) sweep run --recipe recipes/race/race_smoke.yaml --output-dir triage_results/repro/race --strict

7. Check layer checksum mismatches in matrix output

RUN_DIR="$(find triage_results/repro/race -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "layer_checksum_mismatch" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'layer_checksum_mismatches' in metrics:
            values.append(metrics['layer_checksum_mismatches'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

layer_checksum_mismatches must be 0. Non-zero values indicate detected races or silent corruption in distributed layers.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.

Before you start

• Eight-GPU node — amplifies timing races vs the 2-GPU smoke
• Export AORTA_TRIAL_MASTER_PORT_BASE on static launchers (see README-running-recipes)

1. Check out the dashboard commit and install the matching AORTA wheel

git clone https://github.com/ROCm/aorta.git

cd aorta

git checkout 4b4553ef14af08e7ee2d68a888b68cd6a88803f7

python3 -m pip install --upgrade pip

python3 -m pip install --upgrade --pre 'amd-aorta[hw-queue]==0.3.1rc20260905' -f https://github.com/ROCm/aorta/releases/expanded_assets/dev-wheels

# If that exact wheel is unavailable, pick the closest dev-wheel build

# and compare `aorta --version` with the dashboard header before reproducing.

aorta --help

2. Confirm ROCm + PyTorch see enough GPUs

python3 -c "import torch; n=torch.cuda.device_count(); assert torch.cuda.is_available() and n>0, 'no CUDA/HIP devices'; assert n>=8, f'need 8 GPU(s), have {n}'; print(f'{n} GPU(s), HIP', torch.version.hip)"

3. Export single-node torchrun env (adjust for multi-node)

export NCCL_DEBUG=WARN  # optional, for troubleshooting

export MASTER_ADDR=127.0.0.1

export MASTER_PORT=29500

# Slurm / multi-node: see recipes/README-running-recipes.md

4. Race-specific: reserve trial master ports on static launchers

export AORTA_TRIAL_MASTER_PORT_BASE=30000

5. Validate the recipe YAML (no GPU execution)

aorta sweep run --recipe recipes/race/race_smoke.yaml --dry-run

6. Run the recipe with --strict (same flag as nightly CI)

Run from the repository root after setup completes. A standalone --strict sweep only fails cells that error or never run. Dashboard pass/record/fail comes separately from nightly_eval.py comparing matrix.json against config/ci/regression_baselines.yaml; run that harness to apply the dashboard gates.

torchrun --standalone --nproc_per_node=8 $(which aorta) sweep run --recipe recipes/race/race_smoke.yaml --output-dir triage_results/repro/race_8gpu --strict

7. Confirm zero checksum mismatches at 8-GPU scale

RUN_DIR="$(find triage_results/repro/race_8gpu -mindepth 3 -maxdepth 3 -type d -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2-)"

if [ -z "$RUN_DIR" ] || [ ! -d "$RUN_DIR" ]; then echo "No run directory found"; exit 1; fi

echo "Using run directory: $RUN_DIR"

cat "$RUN_DIR/matrix.md"

python3 -m json.tool "$RUN_DIR/matrix.json" | head -80

if [ -f "$RUN_DIR/perf.md" ]; then head -60 "$RUN_DIR/perf.md"; else echo "(no perf.md)"; fi

# Standalone --strict only catches errored or not-run cells; dashboard pass/record/fail comes from nightly_eval.py separately comparing matrix.json to config/ci/regression_baselines.yaml

grep -m 5 -E "layer_checksum_mismatch" "$RUN_DIR/perf.md" 2>/dev/null || true

python3 - "$RUN_DIR" <<'PY'
import json
from pathlib import Path
import sys

run = Path(sys.argv[1])
doc = json.loads((run / 'matrix.json').read_text(encoding='utf-8'))
out = {}
for cell in doc.get('cells') or []:
    values = []
    for raw in cell.get('trial_paths') or []:
        path = Path(raw)
        if not path.is_absolute() and not path.exists():
            path = run / path
        if path.is_dir():
            path = path / 'result.json'
        if not path.is_file():
            continue
        trial = json.loads(path.read_text(encoding='utf-8'))
        metrics = (trial.get('result') or {}).get('metrics') or {}
        if 'layer_checksum_mismatches' in metrics:
            values.append(metrics['layer_checksum_mismatches'])
    out[cell.get('name', '?')] = values or None
print(json.dumps(out, indent=2))
PY

Success criteria

layer_checksum_mismatches == 0 for every cell. Any failure warrants inspecting per-trial JSON under cells/.

Compare with nightly numbers

Match the aorta, PyTorch, ROCm, and HIP versions shown in the dashboard header when comparing numbers. Small differences on different hardware are expected; regressions vs the blessed baseline are what nightly CI flags.