Implements the CI plan (proposed in
ROCm/aorta#300; docs/ci-plan.md lands
with that PR). This is the “how it works / how to operate it” reference for the
nightly evaluation, dashboard, alerting, baselines, and automated bumps.
| Piece | File |
|---|---|
| Eval matrix (workload x config) | config/ci/nightly_eval_matrix.yaml |
| Expected-outcome baselines | config/ci/regression_baselines.yaml |
| CI dependency lock (exact pins) | config/ci/ci-constraints.txt (generated) |
| Harvester + comparator (pure) | scripts/ci/eval_lib.py |
| Nightly harness | scripts/ci/nightly_eval.py |
| Baseline refresher | scripts/ci/refresh_baselines.py |
| Requirements locker | scripts/ci/lock_requirements.sh |
| Dashboard generator | scripts/ci/gen_dashboard.py |
| Regression alerter | scripts/ci/alert_issue.py |
| Nightly workflow | .github/workflows/nightly-eval.yml |
| Pages build + deploy (landing + dashboard) | .github/workflows/pages.yml |
| Baseline refresh workflow | .github/workflows/refresh-baselines.yml |
| Lock refresh workflow | .github/workflows/lock-requirements.yml |
| Automated bumps | .github/dependabot.yml |
Triggered by workflow_run on “Nightly wheels” success (+ workflow_dispatch).
That fires when the whole wheel workflow completes, so nightly.yml holds only
the wheel job; the chat index is published by its own chat-index-nightly.yml,
which follows the same wheel run in parallel with this one rather than ahead of
it (#486).
rocm-ci-setup).amd-aorta[hw-queue] (constrained by
config/ci/ci-constraints.txt when present).nightly_eval.py runs each matrix entry via aorta sweep run --strict,
harvests matrix.json, and compares each cell to the baselines:
passed),fail — a failed/errored cell (even
with no baseline), a missing/empty matrix.json, a per-entry timeout, a
blessed-baseline breach, or a run that did zero work (all skipped). An
empty baseline file therefore protects against crashes/failures immediately;
it just doesn’t add perf/metric gates until blessed. Results go to
gpu-nightly-results.json with build/ROCm metadata.alert_issue.py): opens/updates one nightly-regression issue on
failure; comments + closes it when green.publish job on ubuntu-latest): appends
results/<date>.json to the ci-results data branch (history only).
Only a run triggered by Nightly wheels writes that date key. A
workflow_dispatch has no upstream wheel run (build.upstream_run_id is
empty) and publishes to results/dispatch/<date>-<run_id>.json instead, so
it can never replace that night’s scheduled record, and the dashboard’s
per-day series does not see it.Deploy (pages.yml): a repo has a single Pages site, shared with the
project docs, so one workflow owns the deploy. On main pushes, after each
Nightly Evaluation completes, and on demand, pages.yml builds the Jekyll
site into _site/, relocates the rendered README from _site/index.html to
_site/docs/index.html, writes the self-contained dashboard
(gen_dashboard.py, from the ci-results history) to _site/index.html, and
deploys the combined site via actions/upload-pages-artifact +
actions/deploy-pages. Repo Pages source must be “GitHub Actions”
(Settings -> Pages). Nightly dashboard: https://rocm.github.io/aorta/;
project docs: https://rocm.github.io/aorta/docs/.
Sanitizer nightly dashboard: https://rocm.github.io/aorta/sanitizers/ (linked
from the root nav). The route is always published so it never 404s: before the
first successful sanitizer nightly it shows a “no sanitizer runs yet” empty
state, and if the latest nightly failed it shows the last data under a red
stale banner linking the failed run (rather than silently re-serving the
previous green page). Only sanitizer data produced by a main run is
published.
Each nightly’s per-case results are co-located on the sanitizer-results data
branch under dashboard/runs/<YYYY-MM-DD>T<HHMMSS>-<run_id>/ (one directory per
guardrail recipe, one under survey/ per observed-only case, plus a
meta.json with commit / date / gpu / run_url / gate, where date is an
ISO-8601 instant – runs published before the directory carried a time keep
their date-only names and manifests, and are never renamed), and the rendered page
links to them: the latest run table has a per-recipe Report link, each
kernel-detail card carries a run area link to the case’s directory, and
each Run row in the history table links to its runs/<id>/ area. A
runs/<id>/index.html landing page lists that run’s guardrail reports and its
survey cases. Because pages.yml copies dashboard/* recursively into
_site/sanitizers/, everything under runs/ is served at
/sanitizers/runs/... with relative links and no change to pages.yml. The
render job keeps a rolling window of the newest 30 run directories
(gen_sanitizer_dashboard.py --history-root <dir> --keep 30); older ones are
pruned. Previously these raw reports lived only in an expiring Actions
artifact and were never linked; the rolling window makes them durable and
reachable from the page.
Run areas (#384). A case directory is not just its report – it carries the recipe as it ran, the logs the verdict came from, the source-level inputs, and – for the artifacts it deliberately does not publish – a rebuild command wherever the module’s tables know one (see below), so the copy-paste command the dashboard shows is actionable:
| File | What it is |
|---|---|
index.html |
Landing page: the command, the env a reproduction needs, run identity, observed verdict, the recorded digests (including each code_object_sha256), the fixture rebuild steps, and every file below as a download. GitHub Pages does not auto-index a directory, so this is what makes the run-area link resolve. |
sanitizer_report.json |
The full aorta.sanitizer_report/0.1 document the dashboard renders from. |
consan/consan.log.gz, waitcheck/waitcheck-*.log.gz |
The sanitizer output the verdict was derived from, gzipped. |
recipe.yaml |
The recipe exactly as it ran, pinned to this run. |
inputs/ |
The recipe’s source-level fixture inputs (the .hip repro sources, the GEMM shape CSV) – a few KB in total. |
REPRODUCE.md |
Commit, command, required env, fixture rebuild steps, and the digests of the artifacts that are not published. |
env.json |
The same provenance, machine-readable (aorta.sanitizer_run_area/0.1), including required_env (each variable, who sets it, and why – ConSan’s four are listed only for a case that ran ConSan) and rebuild (the per-artifact commands, runnable from the repo root). REPRODUCE.md and the landing page are rendered from those two stored fields, so the prose cannot disagree with them and a retained area keeps describing what it actually recorded. |
CI-built artifacts are deliberately not published: a GEMM .hsaco runs to
hundreds of MB, and shipping one per retained run would bloat the data branch and
Pages. index.html / REPRODUCE.md / env.json instead record each one’s
path, the SHA-256 the report carries for it (the waitcheck binary digest, the
ConSan repro command and hook digests, every kernel’s code_object_sha256)
and the command that rebuilds it, so a local rebuild can be verified. Neither
is universal: a digest is only recorded when the report has one under that
artifact’s basename, so a bare isa_dir: fixtures/isa reference has none, and
a reference the module’s rebuild tables do not recognise is named without a
command rather than given a plausible-looking guess.
Since that means the file list is not the whole recipe – for a
source.kind: kernel recipe the one input the recipe names is exactly the
excluded one – the landing page’s Files caption says how many inputs are held
back and links to the Artifacts not published table that lists them, naming
their SHA-256s only when every entry has one. The rebuild commands are their
own section higher up the page, so the caption does not promise them under
that anchor. Either way a reader can tell a deliberate omission from a lost
file.
Those rebuild commands are per-artifact, because they are not
interchangeable: a --genco code object is a raw ELF on some ROCm builds and
a clang-offload bundle on others, so it must be unbundled conditionally (the
recorded digest is of the unbundled object the loader opens); the GEMM objects
are extracted from the shipped Tensile libraries by prepare_gemm_isa.py
rather than compiled; and each consan_load / lds_dispatch binary needs the
-DOBJECT / -DLDS_HSACO define naming the object it loads. A generic
“hipcc it” hint would build a different file and fail the digest check.
The commands are also written to be pasted as-is from the repo root, which is
where the git clone in REPRODUCE.md leaves you: fixture paths are rooted at
recipes/sanitizers/fixtures/... (there is no top-level fixtures/), each one
starts by putting the ROCm LLVM bindir on PATH (the container exports only
/opt/rocm/bin, while clang-offload-bundler – which hipcc and
prepare_gemm_isa.py both need – lives beside the compilers) and creating the
gitignored output directory, and the conditional unbundle is a single
if ... else cp ... fi command rather than a prose aside, so an automated
consumer of rebuild can execute the list without interpreting it.
Logs, the recipe copy and its inputs are kept only for the newest 7 runs
(--keep-logs 7) – guardrail and survey areas alike – while reports stay for
the full 30. An older run keeps its report, manifests and landing page, and
both are re-rendered when it is pruned so the page lists only the files still
present and env.json stops naming the inputs it no longer carries.
The dashboard previously lived at /ci/, so that path is kept: /ci/
redirects to the root and /ci/data.json is published alongside
/data.json for anything already polling it. A verification step fails the
deploy if any of those routes would be missing, since a Pages deploy
replaces the whole site and a dropped route 404s immediately.
Three views, ordered so the page reads from “what should I do?” down to detail:
_MOVE_PCT (10%). A quiet night says so
in one line rather than rendering an empty list.Both of the first two withhold themselves until there are two runs to compare; with a single run they would imply a trend from one sample. The page needs no JavaScript – the only script expands and collapses the metric rows, and its controls stay hidden unless it runs.
fail.
Default baselines also bless correctness metrics (exact-equality checksums
such as logits_checksum), so a finite-but-wrong output is caught even without
perf gating. Baselines honor the expected passed outcome (an expected-failure
baseline is supported).tokenspeed_serve_smoke gates
median_tpot_ms and p99_itl_ms on both its cells and nothing else; see
tokenspeed-gating-rollout.md. Every other
entry is still correctness-only.
To turn on perf gating (Phase 5), regenerate baselines with
refresh_baselines.py --perf-gate (adds step_time_ms.max plus per-metric
policy/value bounds – min for throughput, max for latency/step-time, equal
for checksums – that the comparator then enforces; a required metric that is
absent is a failure). --perf-gate-entry <name> (repeatable) restricts that to
named entries: the baseline file is rewritten whole, so an unscoped refresh
arms every entry’s perf gates from whichever single run it just did.
Out-of-scope entries have their correctness data refreshed and their existing
performance bounds carried over untouched, so per-workload rollout is
additive – blessing one workload cannot disarm another. The same holds in the
default mode: omitting --perf-gate does not disarm anything either. An
unscoped --perf-gate re-derives every cell it can, but even that never
deletes a bound it cannot reproduce: a hand-written spec on a _NO_AUTO_GATE
metric (median_itl_ms) or on a metric off the allowlist survives a re-bless
of its own cell, because --perf-gate has nothing to put back in its place.
A refresh may decline to arm a gate; it never disarms one.
Both are dispatch inputs on refresh-baselines.yml (perf_gate and
perf_gate_entry, the latter taking a comma- or space-separated list), so a
scoped refresh is runnable from the Actions UI without editing the workflow.
A perf_gate_entry that is non-empty but names nothing (,, whitespace) is
rejected rather than degrading into the unscoped refresh.
A worked example of rolling gating out for one workload – which metrics to
bound, how many record-only runs to take first, and what to derive the
threshold from – is in
tokenspeed-gating-rollout.md.Baselines are ROCm/stack-specific. Generate them on the runner and bless via PR:
Actions -> Refresh baselines -> Run workflow
refresh_baselines.py runs the matrix, captures each passing cell, and opens a
PR updating regression_baselines.yaml. Review the diff and merge to bless. An
empty baseline file means record-only (nightly won’t be red before blessing).
The refresh fails atomically if any entry ran but couldn’t be blessed
(timeout / missing or empty matrix.json / a cell that didn’t pass) — this
prevents silently reverting live gates to record-only. Entries the runner
can’t exercise at all are not fatal: their existing baselines are
carried over unchanged, so you can still refresh single-GPU baselines on a small
runner and only re-bless the multi-GPU entries on an 8-GPU box. Two declared
capabilities behave this way, and both are honoured on the refresh path as well
as by the nightly:
min_gpus — e.g. the min_gpus: 8 variants on a smaller box.needs_docker_daemon — an entry whose workload starts its own container. The
refresh lane mounts no docker socket, so such an entry skips there. Naming it
in --perf-gate-entry is a hard error rather than a skip: a refresh cannot
derive a bound from a cell it did not run, and a correctness-only file reads in
the PR diff exactly like a successful bless..github/dependabot.yml) opens weekly PRs for pip / docker
(ROCm base image digest) / github-actions updates.config/ci/ci-constraints.txt (partial pip
constraints, not a hash-pinned lock). Both open PRs; a human blesses.
No auto-merge.The publish step keeps only the most recent 180 results/<date>.json files on
the ci-results data branch (older ones are pruned), and the dashboard renders at
most the last 180 builds (gen_dashboard.py --max-builds). Files are tiny; adjust
the cap in nightly-eval.yml / the flag if a longer window is wanted.
Dispatched records under results/dispatch/ have their own 180-file window and
do not count against the scheduled one.
The sanitizer nightly keeps its own rolling window on the sanitizer-results
data branch, and it has two bounds rather than one:
dashboard/runs/<id>/ holds one directory per
guardrail recipe, one under survey/ per observed-only case, and a meta.json.
Each case directory is a run area (report, gzipped logs, recipe.yaml,
inputs/, REPRODUCE.md, env.json, index.html) — see the run-area table
earlier in this document. The render job prunes older runs and re-renders with
gen_sanitizer_dashboard.py --history-root dashboard/runs --keep 30.--keep-logs 7 bounds the logs, the recipe copy and
the copied inputs, for guardrail and survey areas alike. An older run keeps its
report, manifests and landing page; its bulk is pruned and the area is
re-rendered so the page lists only what is still there. Pruning is one-way —
raising --keep-logs later does not bring deleted logs back.Adjust keep in sanitizers-nightly.yml (and the matching --keep) for the
report window, and --keep-logs for the bulk window.
Both windows bound the checkout, not the branch history. Pruning deletes a
file from the working tree, but the blob stays reachable from the commit that
added it, so every log ever published remains in .git even after its area is
pruned — sanitizer-results accumulates ordinary commits indefinitely. Day to
day that costs nothing (the render and push jobs clone --depth 1 and Pages deploys from
the workspace copy), so the exposure is remote repository size. It does change the
deadline for one decision, though: if real ConSan logs turn out large, capping
their size is a code change before the first publish and a history rewrite
afterwards. Measure them on the first nightly (du -sh the staged case dirs in
the render job) rather than waiting for the branch to grow.
The sanitizer dashboard is only ever published from main, so a PR that changes
how it renders would otherwise be seen for the first time after merge. Two
manual paths show it first. Neither runs on its own, and neither writes
sanitizer-results or deploys Pages.
Rendering changes: “Sanitizer dashboard preview” (sanitizer-dashboard-preview.yml).
A few minutes, CPU only. Push the branch, then either Actions -> Sanitizer
dashboard preview -> Use workflow from <branch> -> Run workflow, or:
gh workflow run sanitizer-dashboard-preview.yml --ref <branch>
It clones sanitizer-results read-only and re-renders the published history
with the branch’s generator and baselines, adds the ASan tab the way
pages.yml does, and screenshots the result. The run’s summary page starts with
a PREVIEW – not published header, links the artifact
sanitizer-dashboard-preview-<run_id>, and then shows the rendered
summary.md. The artifact holds dashboard/ (open dashboard/index.html
locally) and screenshots/: dashboard.png, one dashboard-<tab>.png per
further tab, and latest_run.png for the newest run’s landing page. The page
itself carries no preview marking, because history mode shows each run’s own
meta.json. Treat the newest run’s area as approximate: it is re-rendered from
the branch (recipe copies, REPRODUCE.md, env.json), with the container
image and rocjitsu bundle carried over from what that run recorded. The
preview passes no survey artifact of its own. The generator then fills the
Workload survey tab from the newest retained run’s published survey/
directory, so the tab shows that nightly’s observations.
GitHub can only dispatch a workflow file that already exists on main. The PR
that adds or renames this workflow can’t use it, so verify that PR with the
local commands below.
Changes to what the GPU jobs produce: “Sanitizers Nightly” on the branch.
A PR that changes recipes, fixtures or the GPU jobs themselves needs real
reports, which a re-render cannot make. Dispatch Sanitizers Nightly on the
branch (gh workflow run sanitizers-nightly.yml --ref <branch>). Its render
job holds only contents: read. It uploads the rendered tree as
sanitizer-dashboard-<run_id> and, on a non-main ref, screenshots as
sanitizer-dashboard-screenshots-<run_id>. Only the push job writes
sanitizer-results, and it runs on main alone. The GPU runner is shared, so
use this path only when the CPU preview cannot show the change.
On main, that push refuses when sanitizer-results has moved since the render
job recorded its base. Recover by re-running the render job: that re-runs the
dependent push, and render reads the gate artifact already uploaded. Re-running
push alone would commit the stale render. Re-running every job also re-runs the
GPU job on the shared runner and publishes different reports.
Locally, the same steps the preview workflow runs, from a checkout of the branch:
tmp="$(mktemp -d)"
git clone --depth 1 --branch sanitizer-results https://github.com/ROCm/aorta.git "$tmp"
python scripts/sanitizers/gen_sanitizer_dashboard.py \
--history-root "$tmp/dashboard/runs" --keep 30 --keep-logs 7 \
--baselines recipes/sanitizers/fixtures/expected/verdict_baselines.json \
--status "$tmp/dashboard/status.json" \
--out-dir preview/dashboard
python scripts/sanitizers/add_asan_tab.py \
--results docs/sanitizers/asan-manual-runs.json --page preview/dashboard/index.html
pip install "playwright==1.64.1" && python -m playwright install --only-shell chromium
python scripts/sanitizers/screenshot_dashboard.py \
--dashboard-dir preview/dashboard --out preview/screenshots
Locally the newest run’s area loses its container image and rocjitsu bundle
lines unless AORTA_CI_IMAGE, AORTA_ROCJITSU_COMMIT and
AORTA_ROCJITSU_RUN_URL are exported first. The workflow sets them from that
run’s env.json. The same command fills Workload survey from
$tmp/dashboard/runs/<newest>/survey/; do not delete those directories before
rendering.
pages.yml, which serves
the dashboard at / and the project docs under /docs/ from one deploy.
Run the Pages (landing + nightly dashboard) workflow once to publish
immediately (the docs are served even before any nightly results, and the
root shows the dashboard’s empty state).recording rather than passing, because
nothing has been graded against a baseline yet.refresh_baselines.py --perf-gate.