Sanitizers Nightly

← back to CI dashboard

How to read this page

Tools

WaitCheck

Finds s_waitcnt synchronization hazards by static analysis.

ConSan

Finds data races and concurrency issues while a kernel runs.

AddressSanitizer

Reports memory errors. These runs are committed by hand.

Tabs

Nightly gate

Unmatched fails the nightly. A matched fail on daily-consan-racy is the expected control.

Workload survey

Observed only. A fail here does not change the nightly, and a missing scan is marked report missing.

Manual runs

Results the nightly image cannot produce yet. They do not move the Healthy pill.

This run

Run2026-10-31T140437-41200155808

Commitd22f6116235c

Date2026-10-31 14:04:37 UTC

Targetgfx950

Nightly gate Workload survey Manual runs

Latest run 3 recipes

Recipe	Backend	Observed	Expected	Execution	Findings	Coverage	Report	Baseline
daily-waitcheck-gemm	waitcheck (static)	warn	warn	complete	64	—	report	matched
daily-consan-clean	consan (dynamic)	pass	pass	complete	0	0/0, 2/2	report	matched
daily-consan-racy	consan (dynamic)	fail	fail	complete	1	0/0, 2/2	report	matched

Kernel details 3 recipes

daily-waitcheck-gemmwaitcheck (static)warn64 findingsmatched

ObservedwarnExpectedwarnBackendrj_waitcheck 2c406bcc3be3Selectiontop_dispatch_count top‑3Kernels3Executioncomplete

Observationwaitcheck warn; 64 finding(s) (wait_hazard)

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
gemm_NT_M256_N4096_K1024	479	warn	32	sol_126578.hsaco	57c5d8efa4	—	report
gemm_NT_M128_N4096_K1280	471	warn	0	sol_175415.hsaco	57c5d8efa4	same code object as gemm_NT_M256_N4096_K1024; scanned once	report
gemm_TT_M64_N64_K1280	440	warn	32	sol_137678.hsaco	aeb46fded1	—	report

Findings

Sanitizer	Code	Severity	Count	Example
waitcheck	wait_hazard	warning	64	sol_126578.hsaco:gfx950[0]:.text+0x293c: missing s_waitcnt lgkmcnt(14) before use of v20

run area

daily-consan-cleanconsan (dynamic)passno findingsmatched

ObservedpassExpectedpassBackend—Selectiontop_dispatch_count top‑1Kernels1Executioncomplete

Observationconsan pass; preflight pass

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
consan_lds_race	1	pass	0	—	—	—	report

Findings

Sanitizer	Code	Severity	Count	Example
no findings

run area

daily-consan-racyconsan (dynamic)fail1 findingmatched

ObservedfailExpectedfailBackend—Selectiontop_dispatch_count top‑1Kernels1Executioncomplete

Observationconsan fail; 1 finding(s) (sampled_conflict); preflight pass

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
consan_lds_race_2wave	1	fail	1	—	—	—	report

Findings

Sanitizer	Code	Severity	Count	Example
consan	sampled_conflict	race	1	[rocjitsu-dbi-hooks] ConSan conflict reader=100406661497440 first_index=5 second_index=10 first_kind=2 second_kind=1 first_owner=0 second_owner=1 epoch=0 gener…

run area

History / trend 30 runs latest: healthy

Run	Commit	Date	daily-waitcheck-gemm	daily-consan-clean	daily-consan-racy	Gate
2026-10-31T140437-41200155808	d22f6116235c	2026-10-31 14:04:37 UTC	warn matched	pass matched	fail matched	healthy
2026-10-30T141145-41054343941	b78967101935	2026-10-30 14:11:45 UTC	warn matched	pass matched	fail matched	healthy
2026-10-29T141105-40872193672	b78967101935	2026-10-29 14:11:05 UTC	warn matched	pass matched	fail matched	healthy
2026-10-28T140907-40686657660	4fb536db06a8	2026-10-28 14:09:07 UTC	warn matched	pass matched	fail matched	healthy
2026-10-27T155149-40503613953	4fb536db06a8	2026-10-27 15:51:49 UTC	warn matched	pass matched	fail matched	healthy
2026-10-27T135759-40500801889	71099ef28e19	2026-10-27 13:57:59 UTC	warn matched	pass matched	fail matched	healthy
2026-10-26T135357-40333939608	53636046fa47	2026-10-26 13:53:57 UTC	warn matched	pass matched	fail matched	healthy
2026-10-25T122557-40192731938	a347a262ea51	2026-10-25 12:25:57 UTC	warn matched	pass matched	fail matched	healthy
2026-10-24T122537-40093556970	a347a262ea51	2026-10-24 12:25:37 UTC	warn matched	pass matched	fail matched	healthy
2026-10-23T122811-39952936160	7d567f0262b1	2026-10-23 12:28:11 UTC	warn matched	pass matched	fail matched	healthy
2026-10-22T175155-39822417864	7d567f0262b1	2026-10-22 17:51:55 UTC	warn matched	pass matched	fail matched	healthy
2026-10-22T141431-39778180398	3c6a345ae84f	2026-10-22 14:14:31 UTC	warn matched	pass matched	fail matched	healthy
2026-10-21T141345-39599326762	3c6a345ae84f	2026-10-21 14:13:45 UTC	warn matched	pass matched	fail matched	healthy
2026-10-20T141440-39422086566	3c6a345ae84f	2026-10-20 14:14:40 UTC	warn matched	pass matched	fail matched	healthy
2026-10-19T141550-39252050781	3c6a345ae84f	2026-10-19 14:15:50 UTC	warn matched	pass matched	fail matched	healthy
2026-10-18T141043-39126948533	3c6a345ae84f	2026-10-18 14:10:43 UTC	warn matched	pass matched	fail matched	healthy
2026-10-17T140956-39032141956	3c6a345ae84f	2026-10-17 14:09:56 UTC	warn matched	pass matched	fail matched	healthy
2026-10-16T160650-38922506989	3c6a345ae84f	2026-10-16 16:06:50 UTC	warn matched	pass matched	fail matched	healthy
2026-10-16T123432-38900327093	bf117600ca07	2026-10-16 12:34:32 UTC	warn matched	error expected pass unmatched	error expected fail unmatched	failed
2026-10-15T123427-38734461239	bf117600ca07	2026-10-15 12:34:27 UTC	warn matched	error expected pass unmatched	error expected fail unmatched	failed
2026-10-14T195118-38563696959	bf117600ca07	2026-10-14 19:51:18 UTC	warn matched	error expected pass unmatched	error expected fail unmatched	failed
2026-10-14T194155-38397494175	bf117600ca07	2026-10-14 19:41:55 UTC	— report missing	— report missing	— report missing	failed
2026-10-12T142744-38250297996	bf117600ca07	2026-10-12 14:27:44 UTC	warn matched	pass matched	fail matched	healthy
2026-10-11T141417-38139192341	224d19f4a371	2026-10-11 14:14:17 UTC	warn matched	pass matched	fail matched	healthy
2026-10-10T141315-38051823495	224d19f4a371	2026-10-10 14:13:15 UTC	warn matched	pass matched	fail matched	healthy
2026-10-09T141446-37929907291	224d19f4a371	2026-10-09 14:14:46 UTC	warn matched	pass matched	fail matched	healthy
2026-10-08T141657-37776980403	224d19f4a371	2026-10-08 14:16:57 UTC	warn matched	pass matched	fail matched	healthy
2026-10-07T142601-37621140131	cab08a1abc4c	2026-10-07 14:26:01 UTC	warn matched	pass matched	fail matched	healthy
2026-10-06T141219-37463222089	1f59e2040535	2026-10-06 14:12:19 UTC	warn matched	pass matched	fail matched	healthy
2026-10-05T163143-37309782694	b306c32cf2b8	2026-10-05 16:31:43 UTC	warn matched	error expected pass unmatched	error expected fail unmatched	failed

Survey summary

Surveyed 3 workloads (59 kernels) across 6 sanitizer runs2 pass1 warn1 fail1 error1 not_checked

Workload	Kernels	WaitCheck	ConSan	Findings	Note
gemm	1	warn	error	32	combined_hook_timeout
gpt oss	55	pass	fail	43	sampled_conflict
qwen3	3	pass	not_checked	0	1 of 2 kernels not checked (1x consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object)

gemm2 sanitizer runs

WaitCheckstatic wait-count scanwarn32 findings

Backendrj_waitcheck 2c406bcc3be3Selectiontop_dispatch_count top‑1Kernels1Executioncomplete

Observationwaitcheck warn; 32 finding(s) (wait_hazard)

Findingconsan_gemm_f32.hsaco:gfx950[0]:.text+0x293c: missing s_waitcnt lgkmcnt(14) before use of v20

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-waitcheck-gemm-object.yaml

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
gemm_f32_ss	1	warn	32	consan_gemm_f32.hsaco	57c5d8efa4	—	report

Findings

Sanitizer	Code	Severity	Count	Example
waitcheck	wait_hazard	warning	32	consan_gemm_f32.hsaco:gfx950[0]:.text+0x293c: missing s_waitcnt lgkmcnt(14) before use of v20

run area

ConSandynamic data-race checkerrorno findings

Backend—Selectiontop_dispatch_count top‑1Kernels1Executionerror

Observationconsan error; reason combined_hook_timeout; gemm_f32_ss: combined_hook_timeout; preflight error

Reasoncombined_hook_timeout — gemm_f32_ss: combined_hook_timeout

Reproduce

aorta sweep run --recipe recipes/sanitizers/daily-consan-gemm.yaml

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
gemm_f32_ss	1	error	0	consan_gemm_f32.hsaco	57c5d8efa4	combined_hook_timeout	report

Findings

Sanitizer	Code	Severity	Count	Example
no findings

run area

gpt oss2 sanitizer runs

WaitCheckstatic wait-count scanpassno findings

Backendrj_waitcheck 2c406bcc3be3Selectiontop_dispatch_count top‑182Kernels55Executioncomplete

Observationwaitcheck pass

Reproduce

python scripts/sanitizers/model_survey.py run --inventory recipes/sanitizers/models/gpt-oss.yaml --out results

Kernels

Kernel	Dispatch	Observed	Code object	SHA-256	Detail	Report
_matmul_ogs_NNT_bf16xbf16xmxfp4_32x256x128x1_swiglu	1	pass	dispatched.hsaco	02170daa86	—	report
_rope_kernel_cached_thd_2c_gqa_onehead_fwd	1	pass	dispatched.hsaco	02f9a6e283	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	0429ecf4ed	—	report
kernel_unified_attention_2d	1	pass	dispatched.hsaco	05e205ab8f	—	report
_stage2_pow2	1	pass	dispatched.hsaco	1192e120f5	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	136cc91e21	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	1dcf3e3765	—	report
kernel_unified_attention_2d	1	pass	dispatched.hsaco	1e85e67552	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	26b4c78f52	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	28c7fc9948	—	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_64x256x128x1	1	pass	dispatched.hsaco	357789577b	—	report
_rope_kernel_cached_thd_2c_gqa_fwd	1	pass	dispatched.hsaco	44472db069	—	report
_combine_sampled_and_draft_tokens_kernel	1	pass	dispatched.hsaco	44f4a73c86	—	report
_fused_add_rmsnorm_pad	1	pass	dispatched.hsaco	494126cd1a	—	report
_topk_topp_kernel	1	pass	dispatched.hsaco	50fe4fea9d	—	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_32x256x128x1_swiglu	1	pass	dispatched.hsaco	5111186046	—	report
reduce_segments	1	pass	dispatched.hsaco	53c2bad898	—	report
_compute_slot_mappings_kernel	1	pass	dispatched.hsaco	597c8b991d	—	report
_topk_forward	1	pass	dispatched.hsaco	5c4d4d519d	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	61903965e1	—	report
pack_bitmatrix	1	pass	dispatched.hsaco	62b879e054	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	63fa64b777	—	report
_swiglu_bf16xbf16_16x128	1	pass	dispatched.hsaco	6860ae4cbc	—	report
kernel_unified_attention_2d	1	pass	dispatched.hsaco	699e09c798	—	report
kernel_unified_attention_2d	1	pass	dispatched.hsaco	6a27646a37	—	report
kernel_unified_attention_3d	1	pass	dispatched.hsaco	6d038c64ca	—	report
_prepare_pos_seq_lens_kernel	1	pass	dispatched.hsaco	6eb8b8627f	—	report
_rope_kernel_cached_thd_2c_gqa_onehead_fwd	1	pass	dispatched.hsaco	6f97774106	—	report
kernel_unified_attention_2d	1	pass	dispatched.hsaco	7433030bb8	—	report
reduce_segments	1	pass	dispatched.hsaco	7bcee0d9ae	—	report
pack_bitmatrix	1	pass	dispatched.hsaco	7da1e46e4d	—	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_32x256x128x1	1	pass	dispatched.hsaco	8b64fa68f3	—	report
_rope_kernel_cached_thd_2c_gqa_fwd	1	pass	dispatched.hsaco	8f39bbd1bd	—	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_64x256x128x1_swiglu	1	pass	dispatched.hsaco	99783dba6b	—	report
_masked_compaction	1	pass	dispatched.hsaco	a3e32aef6c	—	report
kernel_unified_attention_2d	1	pass	dispatched.hsaco	a5497ec8d0	—	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_32x256x128x1	1	pass	dispatched.hsaco	b60dc71885	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	b6dcdd21dd	—	report
_topk_topp_kernel	1	pass	dispatched.hsaco	b7b68ee37b	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	b7cbcd5bb8	—	report
_get_num_sampled_and_rejected_kernel	1	pass	dispatched.hsaco	cb7a82decf	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	cca3ceda2f	—	report
_post_update_kernel	1	pass	dispatched.hsaco	d39ae3b1c4	—	report
_topk_forward	1	pass	dispatched.hsaco	d46a6cefb7	—	report
_rope_kernel_cached_thd_2c_gqa_onehead_fwd	1	pass	dispatched.hsaco	d5458dc446	—	report
_fused_add_rmsnorm_pad	1	pass	dispatched.hsaco	dc6e5bea86	—	report
pack_bitmatrix	1	pass	dispatched.hsaco	de085311a4	—	report
_fused_add_rmsnorm_kernel	1	pass	dispatched.hsaco	df48165cc5	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	e189a11509	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	e5d86d5d3e	—	report
_stage2_pow2	1	pass	dispatched.hsaco	e6acd0c33b	—	report
_rms_norm_kernel	1	pass	dispatched.hsaco	ec7c47196f	—	report
kernel_unified_attention_3d	1	pass	dispatched.hsaco	f437d8850a	—	report
_prepare_prefill_inputs_kernel	1	pass	dispatched.hsaco	fa4bd10847	—	report
_gather_block_tables_kernel	1	pass	dispatched.hsaco	ffba6285b6	—	report

Findings

Sanitizer	Code	Severity	Count	Example
no findings

run area

ConSandynamic data-race checkfail43 findings

Backend—Selectiontop_dispatch_count top‑23Kernels23Executioncomplete

Observationconsan fail; _prepare_pos_seq_lens_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _prepare_prefill_inputs_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _rope_kernel_cached_thd_2c_gqa_onehead_fwd: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _combine_sampled_and_draft_tokens_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _compute_slot_mappings_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _gather_block_tables_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _get_num_sampled_and_rejected_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _masked_compaction: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object, _post_update_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object; 43 finding(s) (sampled_conflict); preflight pass

Finding[rocjitsu-dbi-hooks] ConSan conflict reader=1039714512 first_index=16 second_index=16 first_kind=2 second_kind=2 first_owner=3 second_owner=3 epoch=1 generatio…

Reproduce

python scripts/sanitizers/model_survey.py run --inventory recipes/sanitizers/models/gpt-oss.yaml --out results

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
_prepare_pos_seq_lens_kernel	1	not_checked	0	dispatched.hsaco	6eb8b8627f	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_prepare_prefill_inputs_kernel	1	not_checked	0	dispatched.hsaco	fa4bd10847	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_rms_norm_kernel	1	fail	8	dispatched.hsaco	b6dcdd21dd	—	report
_rope_kernel_cached_thd_2c_gqa_fwd	1	pass	0	dispatched.hsaco	44472db069	—	report
_rope_kernel_cached_thd_2c_gqa_onehead_fwd	1	not_checked	0	dispatched.hsaco	d5458dc446	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_stage2_pow2	1	fail	1	dispatched.hsaco	e6acd0c33b	—	report
_swiglu_bf16xbf16_16x128	1	pass	0	dispatched.hsaco	6860ae4cbc	—	report
_topk_forward	1	pass	0	dispatched.hsaco	d46a6cefb7	—	report
_topk_topp_kernel	1	fail	9	dispatched.hsaco	b7b68ee37b	—	report
kernel_unified_attention_2d	1	pass	0	dispatched.hsaco	05e205ab8f	—	report
kernel_unified_attention_3d	1	pass	0	dispatched.hsaco	6d038c64ca	—	report
pack_bitmatrix	1	pass	0	dispatched.hsaco	62b879e054	—	report
reduce_segments	1	fail	9	dispatched.hsaco	7bcee0d9ae	—	report
_combine_sampled_and_draft_tokens_kernel	1	not_checked	0	dispatched.hsaco	44f4a73c86	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_compute_slot_mappings_kernel	1	not_checked	0	dispatched.hsaco	597c8b991d	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_fused_add_rmsnorm_kernel	1	fail	8	dispatched.hsaco	b7cbcd5bb8	—	report
_fused_add_rmsnorm_pad	1	fail	8	dispatched.hsaco	dc6e5bea86	—	report
_gather_block_tables_kernel	1	not_checked	0	dispatched.hsaco	ffba6285b6	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_get_num_sampled_and_rejected_kernel	1	not_checked	0	dispatched.hsaco	cb7a82decf	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_masked_compaction	1	not_checked	0	dispatched.hsaco	a3e32aef6c	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_32x256x128x1_swiglu	1	pass	0	dispatched.hsaco	02170daa86	—	report
_matmul_ogs_NNT_bf16xbf16xmxfp4_32x256x128x1	1	pass	0	dispatched.hsaco	8b64fa68f3	—	report
_post_update_kernel	1	not_checked	0	dispatched.hsaco	d39ae3b1c4	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report

Findings

Sanitizer	Code	Severity	Count	Example
consan	sampled_conflict	race	41	[rocjitsu-dbi-hooks] ConSan conflict reader=1039714512 first_index=16 second_index=16 first_kind=2 second_kind=2 first_owner=3 second_owner=3 epoch=1 generatio…
consan	sampled_conflict_summary	race	2	ConSan conflict count reported by the summary only: reader=815590784 counted 80 with 8 example record(s) logged

run area

qwen32 sanitizer runs

WaitCheckstatic wait-count scanpassno findings

Backendrj_waitcheck 2c406bcc3be3Selectiontop_dispatch_count top‑7Kernels3Executioncomplete

Observationwaitcheck pass

Reproduce

python scripts/sanitizers/model_survey.py run --inventory recipes/sanitizers/models/qwen3.yaml --out results

Kernels

Kernel	Dispatch	Observed	Code object	SHA-256	Detail	Report
_topk_topp_kernel	1	pass	dispatched.hsaco	0673f02020	—	report
_compute_slot_mapping_kernel	1	pass	dispatched.hsaco	2b4432007a	—	report
_fwd_kernel	1	pass	dispatched.hsaco	336ef011aa	—	report

Findings

Sanitizer	Code	Severity	Count	Example
no findings

run area

ConSandynamic data-race checknot_checkedno findings

Backend—Selectiontop_dispatch_count top‑2Kernels2Executionpartial

Observationconsan not_checked; reason 1 of 2 kernels not checked (1x consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object); _compute_slot_mapping_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object; preflight pass

Reason1 of 2 kernels not checked (1x consan_not_applicable: no LDS access, barrier, a… — _compute_slot_mapping_kernel: consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object

Reproduce

python scripts/sanitizers/model_survey.py run --inventory recipes/sanitizers/models/qwen3.yaml --out results

Kernels

Kernel	Dispatch	Observed	Findings	Code object	SHA-256	Detail	Report
_compute_slot_mapping_kernel	1	not_checked	0	dispatched.hsaco	2b4432007a	consan_not_applicable: no LDS access, barrier, atomic or fence sites in any loaded code object	report
_fwd_kernel	1	pass	0	dispatched.hsaco	336ef011aa	—	report

Findings

Sanitizer	Code	Severity	Count	Example
no findings

run area

Manual run 2026-10-22 11 rows

Run by hand on 2026-10-22 on MI350 (gfx950) through aorta run --sanitizer asan, proposed in PR #567 (in review, not yet merged); a manual run, not a nightly lane, so CI neither refreshes it nor gates on it. Stack: ROCm 7.0.2 with its ASan runtime libraries; PyTorch 2.10.0 built from source with -fsanitize=address for gfx950:xnack+. Not checked: hipBLASLt GEMM kernels; stores made with buffer instructions; shared memory (LDS) inside PyTorch's kernels; hipMallocManaged memory, and accesses far outside an allocation.

What ran	Result	Kernels checked	Note
gpu_smoke Minimal single-process PyTorch GPU checkaorta run --workload gpu_smoke --sanitizer asan	pass	3 of 3
training PyTorch training loop: forward, loss, backward, optimizer stepaorta run --workload training --sanitizer asan17 hipBLASLt GEMM kernels (Cijk_*): generated assembly that ASan can't instrument, even in an ASan build of hipBLASLt (measured: with hipBLASLt rebuilt with ASan for gfx950:xnack+, all 17 stay unchecked)	not checked	41 of 58	17 hipBLASLt GEMM kernels outside ASan's reach
inference PyTorch eval-only model execution (offline batch, bf16)aorta run --workload inference --sanitizer asan10 hipBLASLt GEMM kernels (Cijk_*): generated assembly that ASan can't instrument, even in an ASan build of hipBLASLt	not checked	24 of 34	10 hipBLASLt GEMM kernels outside ASan's reach
hrx (static) HIP launch probe: a kernel compiled into the binaryhrx probe=static gpu_arch=gfx950	pass	1 of 1
hrx (module) HIP launch probe: a kernel loaded as a separate code objecthrx probe=module gpu_arch=gfx950	pass	1 of 1
hrx (graph_add) HIP launch probe: a kernel launched through a HIP graphhrx probe=graph_add gpu_arch=gfx950	pass	1 of 1
hrx_perf (triad) STREAM triad bandwidth benchmark over 1M floatshrx_perf bench=triad size=1000000 gpu_arch=gfx950	pass	2 of 2
In-bounds view (no bug planted) A view of exactly its 10-float tensor, then t * 2: the same path as the two PyTorch controls below, with nothing to find	clean	2 of 2	nothing planted, nothing reported
Out-of-bounds read in a PyTorch kernel A 16-float view over a 10-float tensor, then t * 2: PyTorch's elementwise kernel reads past the endheap-buffer-overflow, READ of size 4, 0 bytes past the end of a 40-byte allocation, in at::native::vectorized_elementwise_kernel<4, at::native::AUnaryFunctor<float, float, float, at::native::binary_internal::MulFunctor<float>>, std::array<char*, 2ul>>	caught	2 of 2	heap-buffer-overflow READ
Out-of-bounds write in a PyTorch kernel The same over-long view, then t.fill_(1.0): PyTorch's fill kernel writes past the endheap-buffer-overflow, WRITE of size 4, 0 bytes past the end of a 40-byte allocation, in at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor<float>, std::array<char*, 1ul>>	caught	1 of 1	heap-buffer-overflow WRITE
Off-by-one in a HIP kernel hrx's probe kernel with its bounds check changed from i < n to i <= n, built with ASanheap-buffer-overflow, READ of size 4, 0 bytes past the end of a 4000000-byte allocation, in add100 at hrx_probe_oob.cpp:12:24	caught	1 of 1	heap-buffer-overflow READ

PyTorch's kernels are checked only because PyTorch is built with ASan: with the stock PyTorch wheel the same gpu_smoke run checks 0 of its 3 kernels (measured 2026-10-20), since the wheel ships device code without ASan. aorta run has no option for workload configuration yet, and hrx must be built for gfx950 on MI350, so the hrx rows ran through the same run-mode entry point from Python, with the configuration shown. A pass covers what ASan instruments, not every access.