Omnistat (v1.15.0)
Introduction
Overview
What is Omnistat?
User-mode vs System-level monitoring
Software dependencies
Resource Manager Integration
Metrics Available
ROCm
Resource Manager
Host
Process-based I/O
RAS
Occupancy
xGMI
VCN
Hardware Counters
Kernel Tracing
Network
External
Configuration
Script output format
Example
User-supplied
Annotations
Figure of Merit
Advanced Profiling
Hardware counters
Prerequisites
Performance monitoring privileges
System-mode
User-mode
Counter profiles
Sampling modes
Examples
Reported values
Kernel tracing
Prerequisites
Tuning
Combining counters and tracing
Older ROCm releases
System-mode
Installation
Standard install
Runtime configuration
Test the installation
Enable systemd service
Access restriction configuration
Optional component(s)
Prometheus server
Ansible example
SLURM Integration
Grafana Integration
Grafana server
Installation
Data source
Import dashboards
Example screenshots
Cluster Rack View
YAML Configuration
Generating the Dashboard
Importing into Grafana
User-mode
Installation
Standard Omnistat install
Optional component(s)
Victoria Metrics Server
Configuring user-mode Omnistat
Running Jobs
SLURM example
Flux example
Analysis
Exploring results locally
Video demonstration
Combining Omnistat databases
Exporting time series data
Miscellaneous
Agent Support
The skills
Setup
Prerequisites
Enabling the skills with an agent
Examples
Prompts
Sessions
Example session 1: observational report card
Example session 2: hypothesis-driven analysis
Developer Guide
Example collector addition
Implement the uptime data collector
Register the new collector
Putting it all together
License
Site-Specific Instructions
ORNL
Running jobs on Frontier
Storage
Data Analysis
Additional Features
Vendor Counters
Kernel Tracing
LLNL
Running jobs on Tuolumne
Standard usage
Wrapper usage
Storage
Data Analysis
Omnistat
Index
Index