Running Jobs

To enable user-mode data collection for a specific job, add logic within your job script to start and stop the collection mechanism before and after running your desired application(s). Omnistat includes an omnistat-usermode utility to help automate this process and the examples below highlight the steps for simple SLURM and Flux job scripts. Note that the lines highlighted in yellow need to be customized for the local installation path.

SLURM example

Listing 19 Example SLURM job file using user-mode Omnistat with a 10 second sampling interval

#!/bin/bash
#SBATCH -N 8
#SBATCH -n 16
#SBATCH -t 02:00:00

 export OMNISTAT_CONFIG=/path/to/omnistat.config
 export OMNISTAT_DIR=/path/to/omnistat

 # Beginning of job - start data collector
 ${OMNISTAT_DIR}/omnistat-usermode --start --interval 10

 # Run application(s) as normal
 srun <options> ./a.out

 # End of job -  stop data collection, generate summary and store collected data by jobid
 ${OMNISTAT_DIR}/omnistat-usermode --stopexporters
 ${OMNISTAT_DIR}/omnistat-query --job ${SLURM_JOB_ID} --interval 10
 ${OMNISTAT_DIR}/omnistat-usermode --stopserver
 mv data_prom data_prom_${SLURM_JOB_ID}

Flux example

Listing 20 Example FLUX job file using user-mode Omnistat with a 1 second sampling interval

#!/bin/bash
#flux: -N 8
#flux: -n 16
#flux: -t 2h

jobid=`flux getattr jobid`

export OMNISTAT_CONFIG=/path/to/omnistat.config
export OMNISTAT_DIR=/path/to/omnistat

# Beginning of job - start data collector
${OMNISTAT_DIR}/omnistat-usermode --start --interval 1

# Run application(s) as normal
flux run <options> ./a.out

# End of job -  stop data collection, generate summary and store collected data by jobid
${OMNISTAT_DIR}/omnistat-usermode --stopexporters
${OMNISTAT_DIR}/omnistat-query --job ${jobid} --interval 1
${OMNISTAT_DIR}/omnistat-usermode --stopserver
mv data_prom data_prom.${jobid}

In both examples above, the omnistat-query utility is used at the end of the job to query collected telemetry (prior to shutting down the server) for the assigned jobid. This should embed an ascii summary for the job similar to the report card example mentioned in the Overview directly within the recorded job output.