← back to CI dashboard raw reports
⚠Stale. Latest sanitizer nightly run 36568258934 (2026-09-29 12:55:25 UTC) did not complete successfully (failure) — the data below may be stale. view failed run

Sanitizers Nightly

GPU sanitizer guardrails and workload survey

WaitCheck

Static Analysis

Detects potential s_waitcnt synchronization hazards using static code analysis.

ConSan

Runtime Analysis

Detects runtime data races and concurrency issues during execution.

REGRESSIONinvestigate 2/3 sanitizer outcomes that do not match their baselines

This tab is the regression gate. Baseline status column will determine if regression failed or passed.

Latest run 3 recipes

RecipeBackendBaseline statusObserved ExpectedExecutionFindings CoverageReport
daily-waitcheck-gemmwaitcheck (static)Expected outcomewarnwarncomplete64—report
daily-consan-cleanconsan (dynamic)Unexpected outcomeerrorpasserror0—report
daily-consan-racyconsan (dynamic)Unexpected outcomeerrorfailerror0—report

Kernel details 3 recipes

daily-waitcheck-gemmwaitcheck (static)Expected outcomewarn64 findings
ObservedwarnExpectedwarnBackendrj_waitcheck 3b164fa330e8Selectiontop_dispatch_count top‑3Kernels3Executioncomplete
Observationwaitcheck warn; 64 finding(s) (wait_hazard)

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
gemm_NT_M256_N4096_K1024479warn32sol_126578.hsaco57c5d8efa4—report
gemm_NT_M128_N4096_K1280471warn0sol_175415.hsaco57c5d8efa4same code object as gemm_NT_M256_N4096_K1024; scanned oncereport
gemm_TT_M64_N64_K1280440warn32sol_137678.hsacoaeb46fded1—report

Findings

SanitizerCodeSeverityCountExample
waitcheckwait_hazardwarning64sol_126578.hsaco:gfx950[0]:.text+0x1804: missing s_waitcnt lgkmcnt(14) before use of v42
run area
daily-consan-cleanconsan (dynamic)Unexpected outcomeerrorno findings
ObservederrorExpectedpassBackend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason combined_hook_exit_1; consan_lds_race: combined_hook_exit_1; preflight error

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
consan_lds_race1error0——combined_hook_exit_1report

Findings

SanitizerCodeSeverityCountExample
no findings
run area
daily-consan-racyconsan (dynamic)Unexpected outcomeerrorno findings
ObservederrorExpectedfailBackend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason combined_hook_exit_1; consan_lds_race_2wave: combined_hook_exit_1; preflight error

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
consan_lds_race_2wave1error0——combined_hook_exit_1report

Findings

SanitizerCodeSeverityCountExample
no findings
run area
History / trend 30 runs latest: Regression
RunCommitDatedaily-waitcheck-gemmdaily-consan-cleandaily-consan-racyGate
2026-09-29T125525-36568258934e45a9ae778df2026-09-29 12:55:25 UTCMatch warnMismatch error expected passMismatch error expected failRegression
2026-09-28T132908-36422119599f32e64cf324f2026-09-28 13:29:08 UTCMatch warnMismatch error expected passMismatch error expected failRegression
2026-09-28T131255-36318835944f32e64cf324f2026-09-28 13:12:55 UTCReport missing —Report missing —Report missing —Incomplete
2026-09-26T141847-36241711085f32e64cf324f2026-09-26 14:18:47 UTCMatch warnMatch passMatch failHealthy
2026-09-25T142218-361348404582f68c26d995d2026-09-25 14:22:18 UTCMatch warnMatch passMatch failHealthy
2026-09-24T142227-35998938187fe7f92211b622026-09-24 14:22:27 UTCMatch warnMatch passMatch failHealthy
2026-09-23T142606-358603466941680309d023b2026-09-23 14:26:06 UTCMatch warnMatch passMatch failHealthy
2026-09-22T142217-35726994798c9e41ba7dff22026-09-22 14:22:17 UTCMatch warnMatch passMatch failHealthy
2026-09-21T142115-35599335281ef044c5474dc2026-09-21 14:21:15 UTCMatch warnMatch passMatch failHealthy
2026-09-20T141859-35510413832684b4600dbb52026-09-20 14:18:59 UTCMatch warnMatch passMatch failHealthy
2026-09-19T141910-35442659743684b4600dbb52026-09-19 14:19:10 UTCMatch warnMatch passMatch failHealthy
2026-09-18T142430-35344430257684b4600dbb52026-09-18 14:24:30 UTCMatch warnMatch passMatch failHealthy
2026-09-17T162056-35237696462d2fa5f1645c22026-09-17 16:20:56 UTCMatch warnMatch passMatch failHealthy
2026-09-17T132440-3522100477173409f79a56a2026-09-17 13:24:40 UTCMatch warnMismatch error expected passMismatch error expected failRegression
2026-09-17T112050-3521042204573409f79a56a2026-09-17 11:20:50 UTCMatch warnMismatch error expected passMismatch error expected failRegression
2026-09-17T074922-3518976333573409f79a56a2026-09-17 07:49:22 UTCMismatch error expected warnMismatch error expected passMismatch error expected failRegression
2026-09-16T131107-3509573413673409f79a56a2026-09-16 13:11:07 UTCReport missing —Report missing —Report missing —Incomplete
2026-09-15T130837-3496866843573409f79a56a2026-09-15 13:08:37 UTCMismatch error expected warnMismatch error expected passMismatch error expected failRegression
2026-09-14T124757-3484340713373409f79a56a2026-09-14 12:47:57 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-13T124242-3475689263173409f79a56a2026-09-13 12:42:42 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-12T124242-3469344570273409f79a56a2026-09-12 12:42:42 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-11T124542-3459879007573409f79a56a2026-09-11 12:45:42 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-10T124939-344765486006a8c70da0add2026-09-10 12:49:39 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-09T124656-343508947566a8c70da0add2026-09-09 12:46:56 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-08T124853-34225959457d84bea127a2e2026-09-08 12:48:53 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-07T124249-34121807780d5a8bba0383e2026-09-07 12:42:49 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-06T123828-340328623084b4553ef14af2026-09-06 12:38:28 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-05T123756-339658009084b4553ef14af2026-09-05 12:37:56 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-04T124232-338726606894b4553ef14af2026-09-04 12:42:32 UTCMismatch error expected warnMatch passMatch failRegression
2026-09-03T124320-3375519503190dae9376bd32026-09-03 12:43:20 UTCMismatch error expected warnMatch passMatch failRegression

This tab is observational only. Results do not affect nightly pass/fail status and should be interpreted independently.

What You Can Do Here

Understand kernel behaviorSee how real kernels behave under sanitizers.
Investigate sanitizer findingsReview issues detected by WaitCheck or ConSan.
Reproduce issues locallyUse provided commands to reproduce runs.

How Results Are Produced

1
Workload
Run a workload or test recipe
→
2
Kernel Generated
GPU kernel is compiled / extracted
→
3
WaitCheck Analysis
Static scan for s_waitcnt hazards
→
4
ConSan Analysis
Runtime data-race / concurrency check
→
5
Dashboard Results
Reports shown under each sanitizer
Each result includes the workload/recipe and a copy-paste command to reproduce the run.
Important Notes
  • This tab is informational only and does not affect nightly pass/fail status.
  • Findings represent observed behavior, not regressions.
  • Where both sanitizers produced a report the kernel appears under each; a skipped or missing scan still appears, marked report missing with no verdict.
  • Every case lists the recipe that produced the kernel and a reproducible command.

Survey summary

Surveyed 3 kernels across 6 sanitizer runs2 pass1 warn3 error
KernelWaitCheckConSanFindingsNote
gemmwarnerror32combined_hook_exit_1
lds dispatchpasserror0combined_hook_exit_1
tinypasserror0combined_hook_exit_1

gemm2 sanitizer runs

WaitCheckstatic wait-count scanwarn32 findings
Backendrj_waitcheck 3b164fa330e8Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationwaitcheck warn; 32 finding(s) (wait_hazard)
Findingconsan_gemm_f32.hsaco:gfx950[0]:.text+0x1804: missing s_waitcnt lgkmcnt(14) before use of v42

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-gemm-object.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
gemm_f32_ss1warn32consan_gemm_f32.hsaco57c5d8efa4—report

Findings

SanitizerCodeSeverityCountExample
waitcheckwait_hazardwarning32consan_gemm_f32.hsaco:gfx950[0]:.text+0x1804: missing s_waitcnt lgkmcnt(14) before use of v42
run area
ConSandynamic data-race checkerrorno findings
Backend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason combined_hook_exit_1; gemm_f32_ss: combined_hook_exit_1; preflight error
Reasoncombined_hook_exit_1 — gemm_f32_ss: combined_hook_exit_1

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-gemm.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
gemm_f32_ss1error0consan_gemm_f32.hsaco57c5d8efa4combined_hook_exit_1report

Findings

SanitizerCodeSeverityCountExample
no findings
run area

lds dispatch2 sanitizer runs

WaitCheckstatic wait-count scanpassno findings
Backendrj_waitcheck 3b164fa330e8Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationwaitcheck pass

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-lds-dispatch.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
lds_reduce1pass0lds.hsaco2b5a6bce71—report

Findings

SanitizerCodeSeverityCountExample
no findings
run area
ConSandynamic data-race checkerrorno findings
Backend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason combined_hook_exit_1; lds_reduce: combined_hook_exit_1; preflight error
Reasoncombined_hook_exit_1 — lds_reduce: combined_hook_exit_1

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-lds-dispatch.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
lds_reduce1error0lds.hsaco2b5a6bce71combined_hook_exit_1report

Findings

SanitizerCodeSeverityCountExample
no findings
run area

tiny2 sanitizer runs

WaitCheckstatic wait-count scanpassno findings
Backendrj_waitcheck 3b164fa330e8Selectiontop_dispatch_count top‑1Kernels1Executioncomplete
Observationwaitcheck pass

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-tiny.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
tiny_vecadd1pass0tiny.hsacobcfed796dc—report

Findings

SanitizerCodeSeverityCountExample
no findings
run area
ConSandynamic data-race checkerrorno findings
Backend—Selectiontop_dispatch_count top‑1Kernels1Executionerror
Observationconsan error; reason combined_hook_exit_1; tiny_vecadd: combined_hook_exit_1; preflight error
Reasoncombined_hook_exit_1 — tiny_vecadd: combined_hook_exit_1

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-tiny.yaml

Kernels

KernelDispatchObservedFindingsCode objectSHA-256DetailReport
tiny_vecadd1error0tiny.hsacobcfed796dccombined_hook_exit_1report

Findings

SanitizerCodeSeverityCountExample
no findings
run area