schema_version: 1 mode: sanitizer ticket: DAILY-CONSAN-GEMM description: > Informational (non-gating) gfx950 ConSan run over a real, heavy f32 Type_SS hipBLASLt Tensile code object (generic public content extracted in CI from the synthetic gemm_shapes_unique.csv), driven via the source.consan_command path added in #347. This exercises the guardrail's behavior on large production code objects: whatever ConSan concludes, strict policy must reach a real verdict or fail closed, never a false pass. The observed outcome is whatever the run reports -- known rocjitsu limitations hit by this object are tracked in docs/sanitizers/consan-4112-overlapping-anchor-patches.md. Rendered on the dashboard's Workload survey tab (Tab 2) as an observed-only case; it does NOT gate the nightly. sanitizer_plan: target: gfx950 source: kind: kernel kernel: name: gemm_f32_ss code_object: fixtures/isa/consan_gemm_f32.hsaco code_object_index: 0 consan_command: fixtures/bin/consan_gemm_load consan_log: true scope: kind: kernel selection: requirement: top_dispatch_count top_n: 1 sanitizers: - consan policy: consan_policy: strict on_missing_backend: fail # Raised from the 300s cap of #364, which was chosen only because ROCm/rocm-systems#9964 # made MOI inventory non-terminating: every ceiling produced the same # combined_hook_timeout, so a short one bought the identical row for a fraction of # the nightly. #9964 is fixed, so a ceiling that clears the run is now meaningful. # # Sizing history, because the trap here is subtle and worth recording. Three runs # on db0c47df measured 1283s, 1291s and 1416s, and an earlier revision of this # file chose 2000s as ~40% over the slowest. Every one of those runs ended in the # status=4112 transform rejection of ROCm/rocm-systems#10378 -- the object was # never actually instrumented, so they measured a run that gave up early rather # than the work this case represents. A budget measured against a failing run is # a lower bound, not an estimate. # # #10378 is now fixed (dc7c8e04) and was published in bundle 4227d40fb5. The # downloader selects the newest successful run, not that one, so the nightly has # since moved past it -- run 32967422099 consumed 97c1640b. Re-measured against # 4227d40fb5 on the 16,265,200-byte object, that object transforms cleanly # (outcome=modified-valid, 75978 patches over 68894 access sites) and takes # 3696s and 4152s across two runs -- ~62-69 min -- before ending on strict # require-records at exit 86. 2000s would have guaranteed combined_hook_timeout: # the exact failure this recipe change exists to remove. # # 6000s is ~45% over the slower of the two. That headroom absorbs the spread # between them (both runs emitted an identical 508727 log lines, so the ~12% # difference is host contention, not workload variance). It does NOT absorb the # difference between the object those runs timed and the object CI extracts, # and this is the part earlier revisions of this comment got wrong -- they said # only that a different ROCm "may extract a differently sized object". # # Measured, one object per release, same layout (Ailk_Bjlk heavy SS) and the # same unbundle this recipe consumes. These are POST-UNBUNDLE sizes, and are # much larger than what hipBLASLt installs: on the 10.0 tree the shipped .co # is a zlib-compressed offload bundle (CCOB magic, method 1) measuring # 4,081,640 B, which the gfx950 unbundle inflates ~41.6x. Only that release's # bundle was measured on disk; whether 7.0.2 and 7.2.4 shipped compressed # bundles at all is unchecked (neither image is on the gate host), so do not # carry the ratio back to those rows. What ConSan processes, and what this # ceiling is about, is the expansion, not the shipped artifact. # # ROCm 7.0.2 16,265,080 B 245 kernels 217,634 memory instructions # ROCm 7.2.4 191,935,808 B 777 kernels 2,583,251 memory instructions # ROCm 10.0 169,996,560 B 341 kernels 2,093,009 memory instructions # # Kernel counts corrected -- they were 2x here for several revisions, from # counting .kd symbols across both .dynsym and .symtab (llvm-readelf # --symbols prints both tables, 341 each, hence "682"). The 10.0 row is # re-measured directly: 341 .kd in .dynsym, and unbundling that release's # bundle yields 169,996,760 B, 200 B off the recorded figure. The 7.0.2 and # 7.2.4 counts are the recorded numbers halved, NOT re-measured -- neither # image is on the gate host -- so treat them as corrected-by-inference and # re-measure if either becomes load-bearing. The byte and memory-instruction # columns are unchanged; those were always right. # # The timing above was taken on ROCm 7.0.2.2, on an object of 16,265,200 B # (docs/sanitizers/consan-4112-overlapping-anchor-patches.md records it), # which the 7.0.2 row above matches to within 120 bytes. So the # object CI feeds this case is ~10x the timed one on bytes and ~9.6x on the # memory instructions ConSan discovers access sites from -- and it was ~12x on # the old 7.2.4 base too. The gap is not something the ROCm 10 migration # introduced; the migration is the favourable direction on every axis # measurable off-GPU (ROCm 10 is 0.89x the bytes and 0.81x the memory # instructions of 7.2.4, because 10 splits the libraries per device and CI # selects the CU256/ID75a0 variant rather than one bundle covering every part). # # What that means for this ceiling, stated as the bound it is rather than as a # budget: 4152s was dominated by per-site work -- ~76% hook debug logging, and # 508727 log lines against 68894 discovered access sites is ~7.4 lines per site # -- so a ~10x site count does not fit inside a 1.45x margin. This ceiling is # therefore NOT established to clear the run; it is what stops a non-gating row # burning the survey job. It is left at 6000s deliberately rather than raised: # the sanitizers-survey job caps at timeout-minutes: 120 (7200s), so no ceiling # that could plausibly fit ~10x the timed work also fits the job, and inventing # one would replace a measured number with a guess. Re-sizing needs a real # measurement on the gate host against a ROCm 10 object; until then the first # combined_hook_timeout on this row is expected evidence, not a surprise, and # ROCm/rocm-systems#10686 below is the change that would actually make the case # fit. # # This case runs in the non-gating sanitizers-survey job, not on the gate job -- # a ~69-min observed-only row does not belong in front of the Tab 1 verdict. # # Worth knowing before anyone tunes this again: ~76% of that 4152s is the hook # debug logging, not the instrumentation. The same object at RJ_CONSAN_LOG=1 # completes in 991s with byte-identical results (75978 patches, 68894 sites); # the timed phases differ by ~3%. We cannot simply lower the level -- consan_log # maps to kLogDebug because the coverage cross-check needs the per-site records, # which no lower level emits -- so it is filed upstream as # ROCm/rocm-systems#10686. If that lands, this ceiling can drop to roughly a # quarter of its current value. # # !! STALE AS A BUDGET, 2026-08-27 !! Everything above was measured against a # 16,265,200-byte (15.5 MB) object on ROCm 7.0.2.2. The object CI extracts on # the 7.2.4 base is 191,935,808 bytes (~183 MiB) with 637,823 access ranges -- # 11.8x the bytes and 9.3x the sites -- so these figures no longer describe this # case. See prepare_gemm_isa.py's docstring: the size is a per-release property # of the shipped Tensile libraries, not something this repo pins. # # The ceiling is currently NOT what limits this case. ConSan rejects the object # before instrumenting it, on the patched-image growth policy (needs ~1.39 GiB # against a 400 MiB default), so the run ends at ~730s with exit 92 # status=4112. Raising RJ_CONSAN_MAX_PATCHED_IMAGE_GROWTH_PERCENT past ~778 # would let the transform proceed -- and 6000s would then very likely be too # small, because 9.3x the sites is work no measurement here covers. Re-measure # before assuming this number means anything. # See docs/sanitizers/consan-gemm-patched-image-growth-cap.md. timeout_seconds: 6000 output: report: sanitizer_report.json