Loading…
What the perf
nightly measured, run by run: the
HLO
modules of real models replayed one at a time on a gfx950, on every ROCm stack
in the matrix, against the same pinned openxla/xla commit.
Each point is one nightly, not one day.
The sibling view of presubmit CI health is ci-metrics.
The geometric mean of every module’s time against its own first measurement, so 100 is where the window started and 104 is four percent slower. The typical module: alphafold3 at 1.6 s counts exactly as much as detr at 1.4 ms. Lower is faster.
The same runs, summed instead of averaged — what the whole suite costs, and therefore mostly what the three heavy modules cost (alphafold3, stable diffusion 1.5, sdxl). Read beside the index, not instead of it: a run can move here and not there, which says one big module moved and nothing else did. A run that did not measure the whole suite reports no total at all rather than a smaller one.
Total milliseconds per run, split by model category. One panel per stack, all on one scale so the panels can be read against each other.
One panel per HLO module, one line per ROCm stack. This is the view the two aggregates on the Models tab average away: a suite that holds still can still contain a module that doubled, and only this says which one. Inference HLO is the forward pass only, dumped on one GPU.
Each panel has its own y axis, zoomed to that module's own range — these modules run from under 2 ms to over 1.5 s, so a shared scale would flatten all but the largest into a single line. Panels are therefore comparable with themselves over time, never with each other; the figures at the top and bottom of each are its own highest and lowest readings.
One panel per HLO module, one line per ROCm stack. This is the view the two aggregates on the Models tab average away: a suite that holds still can still contain a module that doubled, and only this says which one. Training HLO is a separate dump and a separate suite: it carries the backward pass, the gradient all-reduce and the optimizer update, none of which inference runs, so a module here has no inference counterpart.
Each panel has its own y axis, zoomed to that module's own range — these modules run from under 2 ms to over 1.5 s, so a shared scale would flatten all but the largest into a single line. Panels are therefore comparable with themselves over time, never with each other; the figures at the top and bottom of each are its own highest and lowest readings.
rocm_latency_check_tool, which times a H2D → busy-kernel → D2D → busy-kernel → D2H round trip against a fixed-duration GPU kernel. The compute is deliberately constant, so what moves here is HIP runtime and copy overhead, not kernel selection. One line per GPU count.
One stack, not three. This figure characterises the ROCm runtime rather than the XLA tree, so the nightly runs the tool on the newest stack only; re-measuring it on the pinned releases would only re-measure the same interconnect. A handful of early runs did measure all three, before that condition was added, and those rows are discarded rather than drawn as two lines that stop in November and never resume. The stack filter above does not apply here, and is hidden on this tab for that reason.
Each figure is one 10-second run per GPU count. The median holds steady; the p99 is a handful of requests and a single stall moves it by an order of magnitude, so read it as a count of stalls rather than as latency.
Every plotted value, as text.