Installation

Quick Start (Latest Release, Binary Installer)

Download the omnitrace-install.py and specify --prefix <install-directory>. This script will attempt to auto-detect the appropriate OS distribution and OS version. If ROCm support is desired, specify --rocm X.Y where X is the ROCm major version and Y is the ROCm minor version, e.g. --rocm 5.4.

wget https://github.com/ROCm/omnitrace/releases/latest/download/omnitrace-install.py
python3 ./omnitrace-install.py --prefix /opt/omnitrace --rocm 5.4

This script supports installation on Ubuntu, OpenSUSE, RedHat, Debian, CentOS, and Fedora. If the target OS is compatible with one of the operating system versions below, specify -d <DISTRO> -v <VERSION>, e.g. if the OS is compatible with Ubuntu 18.04, pass -d ubuntu -v 18.04 to the script.

Operating System

OmniTrace is only supported on Linux. The following distributions are tested:

• Ubuntu 18.04

• Ubuntu 20.04

• Ubuntu 22.04

• OpenSUSE 15.2

• OpenSUSE 15.3

• OpenSUSE 15.4

• RedHat 8.7

• RedHat 9.0

• RedHat 9.1

Other OS distributions may be supported but are not tested.

Identifying the Operating System

If you are unsure of the operating system and version, the /etc/os-release and /usr/lib/os-release files contain operating system identification data for Linux systems.

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
...
VERSION_ID="20.04"
...

The relevent fields are ID and the VERSION_ID.

Architecture

With regards to instrumentation, at present only amd64 (x86_64) architectures are tested; however, Dyninst supports several more architectures and thus, omnitrace instrumentation may support other CPU architectures such as aarch64, ppc64, etc. Other modes of use, such as sampling and causal profiling, are not dependent on Dyninst and therefore may be more portable.

Installing omnitrace from binary distributions

Every omnitrace release provides binary installer scripts of the form:

omnitrace-{VERSION}-{OS_DISTRIB}-{OS_VERSION}[-ROCm-{ROCM_VERSION}[-{EXTRA}]].sh

E.g.:

omnitrace-1.0.0-ubuntu-18.04-OMPT-PAPI-Python3.sh
omnitrace-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI-Python3.sh
...
omnitrace-1.0.0-ubuntu-20.04-ROCm-50000-OMPT-PAPI-Python3.sh

Any of the EXTRA fields with a cmake build option (e.g. PAPI, see below) or no link requirements (e.g. OMPT) have self-contained support for these packages.

Download the appropriate binary distribution

wget https://github.com/ROCm/omnitrace/releases/download/v<VERSION>/<SCRIPT>

Create the target installation directory

mkdir /opt/omnitrace

Run the installer script

./omnitrace-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI.sh --prefix=/opt/omnitrace --exclude-subdir

Installing OmniTrace from source

Build Requirements

OmniTrace needs a GCC compiler with full support for C++17 and CMake v3.16 or higher. The Clang compiler may be used in lieu of the GCC compiler if Dyninst is already installed.

• GCC compiler v7+

  • Older GCC compilers may be supported but are not tested

  • Clang compilers are generally supported for OmniTrace but not Dyninst

• CMake v3.16+

If the system installed cmake is too old, installing a new version of cmake can be done through several methods. One of the easiest options is to use PyPi (i.e. python’s pip):

pip install --user 'cmake==3.18.4'
export PATH=${HOME}/.local/bin:${PATH}

Required Third-Party Packages

• DynInst for dynamic or static instrumentation

  • TBB required by Dyninst

  • ElfUtils required by Dyninst

  • LibIberty required by Dyninst

  • Boost required by Dyninst

  • OpenMP optional by Dyninst

• libunwind for call-stack sampling

All of the third-party packages required by DynInst and DynInst itself can be built and installed during the build of omnitrace itself. In the list below, we list the package, the version, which package requires the package (i.e. omnitrace requires Dyninst and Dyninst requires TBB), and the CMake option to build the package alongside omnitrace:

Third-Party Library	Minimum Version	Required By	CMake Option
Dyninst	12.0	OmniTrace	OMNITRACE_BUILD_DYNINST (default: OFF)
Libunwind		OmniTrace	OMNITRACE_BUILD_LIBUNWIND (default: ON)
TBB	2018.6	Dyninst	DYNINST_BUILD_TBB (default: OFF)
ElfUtils	0.178	Dyninst	DYNINST_BUILD_ELFUTILS (default: OFF)
LibIberty		Dyninst	DYNINST_BUILD_LIBIBERTY (default: OFF)
Boost	1.67.0	Dyninst	DYNINST_BUILD_BOOST (default: OFF)
OpenMP	4.x	Dyninst

Optional Third-Party Packages

• ROCm

  • HIP

  • Roctracer for HIP API and kernel tracing

  • ROCM-SMI for GPU monitoring

  • Rocprofiler for GPU hardware counters

• PAPI

• MPI

  • OMNITRACE_USE_MPI will enable full MPI support

  • OMNITRACE_USE_MPI_HEADERS will enable wrapping of the dynamically-linked MPI C function calls

    • By default, if an OpenMPI MPI distribution cannot be found, omnitrace will use a local copy of the OpenMPI mpi.h

• Several optional third-party profiling tools supported by timemory (e.g. Caliper, TAU, CrayPAT, etc.)

Third-Party Library	CMake Enable Option	CMake Build Option
PAPI	OMNITRACE_USE_PAPI (default: ON)	OMNITRACE_BUILD_PAPI (default: ON)
MPI	OMNITRACE_USE_MPI (default: OFF)
MPI (header-only)	OMNITRACE_USE_MPI_HEADERS (default: ON)

Installing DynInst

Building Dyninst alongside OmniTrace

The easiest way to install Dyninst is to configure omnitrace with OMNITRACE_BUILD_DYNINST=ON. Depending on the version of Ubuntu, the apt package manager may have current enough versions of Dyninst’s Boost, TBB, and LibIberty dependencies (i.e. apt-get install libtbb-dev libiberty-dev libboost-dev); however, it is possible to request Dyninst to install it’s dependencies via DYNINST_BUILD_<DEP>=ON, e.g.:

git clone https://github.com/ROCm/omnitrace.git omnitrace-source
cmake -B omnitrace-build -DOMNITRACE_BUILD_DYNINST=ON -DDYNINST_BUILD_{TBB,ELFUTILS,BOOST,LIBIBERTY}=ON omnitrace-source

where -DDYNINST_BUILD_{TBB,BOOST,ELFUTILS,LIBIBERTY}=ON is expanded by the shell to -DDYNINST_BUILD_TBB=ON -DDYNINST_BUILD_BOOST=ON ...

Installing Dyninst via Spack

Spack is another option to install Dyninst and it’s dependencies:

git clone https://github.com/spack/spack.git
source ./spack/share/spack/setup-env.sh
spack compiler find
spack external find --all --not-buildable
spack spec -I --reuse dyninst
spack install --reuse dyninst
spack load -r dyninst

Installing omnitrace

OmniTrace has cmake configuration options for supporting MPI (OMNITRACE_USE_MPI or OMNITRACE_USE_MPI_HEADERS), HIP kernel tracing (OMNITRACE_USE_ROCTRACER), sampling ROCm devices (OMNITRACE_USE_ROCM_SMI), OpenMP-Tools (OMNITRACE_USE_OMPT), hardware counters via PAPI (OMNITRACE_USE_PAPI), among others. Various additional features can be enabled via the TIMEMORY_USE_* CMake options. Any OMNITRACE_USE_<VAL> option which has a corresponding TIMEMORY_USE_<VAL> option means that the support within timemory for this feature has been integrated into omnitrace’s perfetto support, e.g. OMNITRACE_USE_PAPI=<VAL> forces TIMEMORY_USE_PAPI=<VAL> and the data that timemory is able to collect via this package is passed along to perfetto and will be displayed when the .proto file is visualized in ui.perfetto.dev.

git clone https://github.com/ROCm/omnitrace.git omnitrace-source
cmake                                       \
    -B omnitrace-build                      \
    -D CMAKE_INSTALL_PREFIX=/opt/omnitrace  \
    -D OMNITRACE_USE_HIP=ON                 \
    -D OMNITRACE_USE_ROCM_SMI=ON            \
    -D OMNITRACE_USE_ROCTRACER=ON           \
    -D OMNITRACE_USE_PYTHON=ON              \
    -D OMNITRACE_USE_OMPT=ON                \
    -D OMNITRACE_USE_MPI_HEADERS=ON         \
    -D OMNITRACE_BUILD_PAPI=ON              \
    -D OMNITRACE_BUILD_LIBUNWIND=ON         \
    -D OMNITRACE_BUILD_DYNINST=ON           \
    -D DYNINST_BUILD_TBB=ON                 \
    -D DYNINST_BUILD_BOOST=ON               \
    -D DYNINST_BUILD_ELFUTILS=ON            \
    -D DYNINST_BUILD_LIBIBERTY=ON           \
    omnitrace-source
cmake --build omnitrace-build --target all --parallel 8
cmake --build omnitrace-build --target install
source /opt/omnitrace/share/omnitrace/setup-env.sh

MPI Support within OmniTrace

OmniTrace can have full (OMNITRACE_USE_MPI=ON) or partial (OMNITRACE_USE_MPI_HEADERS=ON) MPI support. The only difference between these two modes is whether or not the results collected via timemory and/or perfetto can be aggregated into a single output file during finalization. When full MPI support is enabled, combining the timemory results always occurs whereas combining the perfetto results is configurable via the OMNITRACE_PERFETTO_COMBINE_TRACES setting.

The primary benefits of partial or full MPI support are the automatic wrapping of MPI functions and the ability to label output with suffixes which correspond to the MPI_COMM_WORLD rank ID instead of using the system process identifier (i.e. PID). In general, it is recommended to use partial MPI support with the OpenMPI headers as this is the most portable configuration. If full MPI support is selected, make sure your target application is built against the same MPI distribution as omnitrace, i.e. do not build omnitrace with MPICH and use it on a target application built against OpenMPI. If partial support is selected, the reason the OpenMPI headers are recommended instead of the MPICH headers is because the MPI_COMM_WORLD in OpenMPI is a pointer to ompi_communicator_t (8 bytes), whereas MPI_COMM_WORLD in MPICH, it is an int (4 bytes). Building omnitrace with partial MPI support and the MPICH headers and then using omnitrace on an application built against OpenMPI will cause a segmentation fault due to the value of the MPI_COMM_WORLD being narrowed during the function wrapping before being passed along to the underlying MPI function.

Post-Installation Steps

Configure the environment

If environment modules are available and preferred:

module use /opt/omnitrace/share/modulefiles
module load omnitrace/1.0.0

Alternatively, once can directly source the setup-env.sh script:

source /opt/omnitrace/share/omnitrace/setup-env.sh

Test the executables

Successful execution of these commands indicates that the installation does not have any issues locating the installed libraries:

omnitrace-instrument --help
omnitrace-avail --help

NOTE: If ROCm support was enabled, you may have to add the path to the ROCm libraries to LD_LIBRARY_PATH, e.g. export LD_LIBRARY_PATH=/opt/rocm/lib:${LD_LIBRARY_PATH}