← back to CI dashboard raw reports
⚠Stale. Latest sanitizer nightly run 33965800908 (2026-09-05 12:37:56 UTC) did not complete successfully (failure) — the data below may be stale. view failed run

Sanitizers Nightly

GPU sanitizer guardrails and workload survey

WaitCheck

Static Analysis

Detects potential s_waitcnt synchronization hazards using static code analysis.

ConSan

Runtime Analysis

Detects runtime data races and concurrency issues during execution.

REGRESSIONinvestigate 1/3 sanitizer outcomes that do not match their baselines

This tab is the regression gate. Baseline status column will determine if regression failed or passed.

Latest run 3 recipes

RecipeBackendBaseline statusObserved ExpectedExecutionFindings CoverageReport
daily-waitcheck-gemmwaitcheck (static)Unexpected outcomeerrorwarnerror32—report
daily-consan-cleanconsan (dynamic)Expected outcomepasspasscomplete00/0, 2/2report
daily-consan-racyconsan (dynamic)Expected outcomefailfailcomplete640/0, 2/2report

Kernel details 3 recipes

daily-waitcheck-gemmwaitcheck (static)Unexpected outcomeerror32 findings
ObservederrorExpectedwarnBackendrj_waitcheck a70945fb1135Selectiontop_dispatch_count top‑3Kernels3Executionerror
Observationwaitcheck error; reason worklist_not_fully_checked; 32 finding(s) (wait_hazard)

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
gemm_NT_M256_N4096_K1024479error0sol_126578.hsaco57c5d8efa4report
gemm_NT_M128_N4096_K1280471—0sol_175415.hsaco57c5d8efa4report
gemm_TT_M64_N64_K1280440warn32sol_137678.hsacoaeb46fded1report

Findings

SanitizerCodeSeverityCountExample
waitcheckwait_hazardwarning32sol_137678.hsaco:gfx950[0]:.text+0x4a4: missing s_waitcnt lgkmcnt(0) before def of s45
run area
daily-consan-cleanconsan (dynamic)Expected outcomepassno findings
ObservedpassExpectedpassBackend—Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationconsan pass; preflight pass

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
consan_lds_race1pass0——report

Findings

SanitizerCodeSeverityCountExample
no findings
run area
daily-consan-racyconsan (dynamic)Expected outcomefail64 findings
ObservedfailExpectedfailBackend—Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationconsan fail; 64 finding(s) (1); preflight pass

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
consan_lds_race_2wave1fail64——report

Findings

SanitizerCodeSeverityCountExample
consan1race64[rocjitsu-dbi-hooks] ConSan MOI auto replay diagnostic reader=104792620710192 index=0 kind=1 code_object=fnv1a64:3deb93883c6c6fa8 report_generation=2 generatio…
run area
History / trend 30 runs latest: Regression
RunCommitDatedaily-waitcheck-gemmdaily-consan-cleandaily-consan-racyGate
2026-09-05T123756-339658009084b4553ef14af2026-09-05 12:37:56 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-04T124232-338726606894b4553ef14af2026-09-04 12:42:32 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-03T124320-3375519503190dae9376bd32026-09-03 12:43:20 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-03T103956-3374398059190dae9376bd32026-09-03 10:39:56 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-02T124713-3362990148230dcc055cba82026-09-02 12:47:13 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-01T125503-33507543444d11092a603d92026-09-01 12:55:03 UTCMatch warnMatch passMatch failHealthy
2026-08-31T125804-33391758880c57a1e4b07202026-08-31 12:58:04 UTCMatch warnMatch passMatch failHealthy
2026-08-30T125306-33311286343c57a1e4b07202026-08-30 12:53:06 UTCMatch warnMatch passMatch failHealthy
2026-08-29T124958-33252284140e61f455a130b2026-08-29 12:49:58 UTCMatch warnMatch passMatch failHealthy
2026-08-28T164558-33189127590e61f455a130b2026-08-28 16:45:58 UTCMatch warnMatch passMatch failHealthy
2026-08-27T162009-33089985218e61f455a130b2026-08-27 16:20:09 UTCMatch warnMatch passMatch failHealthy
2026-08-26T124411-32967422099bed2771d52dd2026-08-26 12:44:11 UTCMatch warnMatch passMatch failHealthy
2026-08-25-32846400271905b1f9e3e162026-08-25Match warnMatch passMatch failHealthy
2026-08-24-3272590387878d1ae686dc32026-08-24Match warnMatch passMatch failHealthy
2026-08-23-3263858470478d1ae686dc32026-08-23Match warnMatch passMatch failHealthy
2026-08-22-3257221307778d1ae686dc32026-08-22Match warnMatch passMatch failHealthy
2026-08-21-3248076820178d1ae686dc32026-08-21Match warnMatch passMatch failHealthy
2026-08-20-3237314695278d1ae686dc32026-08-20Match warnMatch passMatch failHealthy
2026-08-20-32367664704b4be48fdd6dd2026-08-20Match warnMatch passMatch failHealthy
2026-08-19-32251405821b4be48fdd6dd2026-08-19Match warnMatch passMatch failHealthy
2026-08-18-32135585222b4be48fdd6dd2026-08-18Match warnMatch passMatch failHealthy
2026-08-17-32028590251b4be48fdd6dd2026-08-17Match warnMatch passMatch failHealthy
2026-08-16-31946286609b4be48fdd6dd2026-08-16Match warnMatch passMatch failHealthy
2026-08-15-31883858931b4be48fdd6dd2026-08-15Match warnMatch passMatch failHealthy
2026-08-14-3180000054668c980aa449f2026-08-14Match warnMatch passMatch failHealthy
2026-08-13-3171850248568c980aa449f2026-08-13Match warnMatch passMatch failHealthy
2026-08-13-3171402608468c980aa449f2026-08-13Match warnMatch passMatch failHealthy
2026-08-13-31699735119f1660c492df62026-08-13Match warnMatch passMatch failHealthy
2026-08-12-31615014466f1660c492df62026-08-12Match warnMatch passMatch failHealthy
2026-08-12-315989067195fbbb5c04dc42026-08-12Match warnMatch passMatch failHealthy

This tab is observational only. Results do not affect nightly pass/fail status and should be interpreted independently.

What You Can Do Here

Understand kernel behaviorSee how real kernels behave under sanitizers.
Investigate sanitizer findingsReview issues detected by WaitCheck or ConSan.
Reproduce issues locallyUse provided commands to reproduce runs.

How Results Are Produced

1
Workload
Run a workload or test recipe
→
2
Kernel Generated
GPU kernel is compiled / extracted
→
3
WaitCheck Analysis
Static scan for s_waitcnt hazards
→
4
ConSan Analysis
Runtime data-race / concurrency check
→
5
Dashboard Results
Reports shown under each sanitizer
Each result includes the workload/recipe and a copy-paste command to reproduce the run.
Important Notes
  • This tab is informational only and does not affect nightly pass/fail status.
  • Findings represent observed behavior, not regressions.
  • Where both sanitizers produced a report the kernel appears under each; a skipped or missing scan still appears, marked report missing with no verdict.
  • Every case lists the recipe that produced the kernel and a reproducible command.

Survey summary

Surveyed 3 kernels across 6 sanitizer runs3 pass3 error
KernelWaitCheckConSanFindingsNote
gemmerrorerror0consan_strict_load_rejection
lds dispatchpasspass0—
tinypasserror0combined_hook_exit_86

gemm2 sanitizer runs

WaitCheckstatic wait-count scanerrorno findings
Backendrj_waitcheck a70945fb1135Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationwaitcheck error; reason worklist_not_fully_checked
Reasonworklist_not_fully_checked

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-gemm-object.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
gemm_f32_ss1error0consan_gemm_f32.hsaco57c5d8efa4report

Findings

SanitizerCodeSeverityCountExample
no findings
run area
ConSandynamic data-race checkerrorno findings
Backend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason consan_strict_load_rejection; preflight error
Reasonconsan_strict_load_rejection

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-gemm.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
gemm_f32_ss1error0consan_gemm_f32.hsaco57c5d8efa4report

Findings

SanitizerCodeSeverityCountExample
no findings
run area

lds dispatch2 sanitizer runs

WaitCheckstatic wait-count scanpassno findings
Backendrj_waitcheck a70945fb1135Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationwaitcheck pass

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-lds-dispatch.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
lds_reduce1pass0lds.hsaco6bae96302areport

Findings

SanitizerCodeSeverityCountExample
no findings
run area
ConSandynamic data-race checkpassno findings
Backend—Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationconsan pass; preflight pass

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-lds-dispatch.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
lds_reduce1pass0lds.hsaco6bae96302areport

Findings

SanitizerCodeSeverityCountExample
no findings
run area

tiny2 sanitizer runs

WaitCheckstatic wait-count scanpassno findings
Backendrj_waitcheck a70945fb1135Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationwaitcheck pass

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-tiny.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
tiny_vecadd1pass0tiny.hsaco7e7e19e27freport

Findings

SanitizerCodeSeverityCountExample
no findings
run area
ConSandynamic data-race checkerrorno findings
Backend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason combined_hook_exit_86; preflight error
Reasoncombined_hook_exit_86

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-tiny.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256Report
tiny_vecadd1error0tiny.hsaco7e7e19e27freport

Findings

SanitizerCodeSeverityCountExample
no findings
run area