User-mode installation

As mentioned in the project overview, Omnistat has two primary modes of operation and this section highlights installation of the user-mode variant which does not require elevated credentials and targets temporary data collection directly within a user’s batch job under a supported resource manager (e.g. SLURM, or Flux).

In user-mode executions, Omnistat data collectors and a companion VictoriaMetrics server are deployed temporarily on hosts assigned to a user’s job, as highlighted in Figure 2. The following assumptions are made throughout the rest of this user-mode installation discussion:

Assumptions:

ROCm v6.3.0 or newer is pre-installed on all GPU hosts.
Installer has access to a distributed file-system; if no distributed file-system is present, installation steps need to be repeated across all nodes.

The steps below install a development version of Omnistat using git for a basic installation that enables standard GPU and host-level telemetry. Depending on your local environment, you may also wish to augment the examples that follow to install Omnistat within a dedicated Python virtual environment (e.g. using venv or conda).

Standard Omnistat install

Use latest development from git source

Clone the repository (dev branch is default) and install python dependencies:

$ git clone https://github.com/ROCm/omnistat.git
$ cd omnistat
$ pip install -r requirements.txt

Optional component(s)

Beyond the standard data collector, Omnistat provides optional components that unlock additional telemetry: a GPU hardware counter collector and a kernel tracing library, both built on ROCProfiler-SDK. These steps are optional and only required to enable support for hardware counter or kernel tracing collection — the standard install above already enables GPU and host-level monitoring. These components are compiled from C++ sources and require a local build step.

Use latest development from git source

From within a cloned copy of the repository, build the optional components in place.

Hardware counter support consists of two pieces. First, build the collector extension that samples counters from the GPUs:

$ pip install cmake-build-extension nanobind
$ BUILD_ROCPROFILER_SDK_EXTENSION=1 python setup.py build_ext --inplace

Then build the counter enablement library (libomnistat_count.so), a standalone C++ shared library loaded into monitored applications to enable counter collection for their queues (required for user-mode collection):

$ cmake -S omnistat/rocprofiler-sdk/ -B build-count/ -DBUILD_COUNT_LIB=ON
$ cmake --build build-count/

Kernel tracing support provides libomnistat_trace.so, a standalone C++ shared library that intercepts GPU kernel dispatches at runtime:

$ cmake -S omnistat/rocprofiler-sdk/ -B build-trace/ -DBUILD_KERNEL_TRACE_LIB=ON
$ cmake --build build-trace/

Note

The optional components rely on cmake and a HIP C++ compiler.

The resulting libraries are located at build-count/libomnistat_count.so and build-trace/libomnistat_trace.so. See Advanced Profiling for usage instructions, covering both hardware counters and kernel tracing.

Victoria Metrics Server

Download a single-node VictoriaMetrics server. Assuming a victoria-metrics server is not already present on the system, download and extract a precompiled binary from upstream. This binary can generally be stored in any directory accessible by the user, but the path to the binary will need to be known during the next section when configuring user-mode execution. Note that VictoriaMetrics provides a larger number of binary releases and we typically use the victoria-metrics-linux-amd64 variant on x86_64 clusters.

Configuring user-mode Omnistat

For user-mode execution, Omnistat includes additional options in the [omnistat.usermode] section of the runtime configuration file. A portion of the default config file is highlighted below with the lines in yellow indicating settings to confirm or customize for your local environment.

Listing 18 Sample Omnistat configuration file

 [omnistat.collectors]
 port = 8001
 enable_amd_smi = True
 enable_rms = True

 [omnistat.collectors.rms]
 job_detection_mode = file-based
 job_detection_file = /tmp/omni_rmsjobinfo_user

 [omnistat.usermode]
 victoria_binary = /path/to/victoria-metrics
 victoria_datadir = data_prom
 victoria_logfile = vic_server.log