Checking Job Efficiency

When slurm grants the resource (CPU, memory, GPU) that you requested for a job, it will reserve (and bill for) that resource while the job is running, regardless of whether the job actually uses the resource or not. Review resource usage while a job is running to identify bottlenecks, and use completed, representative jobs to estimate future resource requests. Running-job accounting can be incomplete, as explained below. Low utilization alone does not identify a problem: a job may be waiting for data, I/O or synchronization.

For a practical investigation, see Troubleshooting Resource Efficiency.

CPU and memory efficiency

The easiest way to check CPU and memory utilization efficiency of a job is to use the seff script:

$ seff <JOBID>

where JOBID is the unique ID of the job(step) or a comma separated list of job(step) IDs.

An example:

$ seff 1234567
Job ID: 1234567
Cluster: komondor
User/Group: alice/alice
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 8
CPU Utilized: 01:08:43
CPU Efficiency: 88.10% of 01:18:00 core-walltime
Job Wall-clock time: 00:09:45
Memory Utilized: 2.74 GB
Memory Efficiency: 68.56% of 4.00 GB

A more detailed and highly customizable report can be generated using the sacct command:

$ sacct -j <JOBID> -o <FIELD1,FIELD2,...>

where FIELD1,FIELD2,… is a comma separated list of the required information.

For example, here is how you can get statistics somewhat similar to the output of the “seff” command using sacct:

$ sacct -j 1234567 -o JobID,Cluster,User,Group,State,ExitCode,NNodes,NCPUs,TotalCPU,Elapsed,MaxRSS,ReqMem
JobID           Cluster      User     Group      State ExitCode   NNodes      NCPUS   TotalCPU    Elapsed     MaxRSS     ReqMem
------------ ---------- --------- --------- ---------- -------- -------- ---------- ---------- ---------- ---------- ----------
3115439        komondor   hpctefo   hpctefo  COMPLETED      0:0        1          8   01:08:43   00:09:45                    4G
3115439.bat+   komondor                      COMPLETED      0:0        1          8   01:08:43   00:09:45   2875752K

Use the --helpformat option for a list of available fields:

$ sacct --helpformat

For detailed information see the manpages for sacct (type man sacct in the terminal).

Note

If the job is still running, some fields (e.g. regarding the memory usage) in the sacct output are not available and some information in the seff output can be incomplete and misleading.

Note

Statistics of interrupted jobs (e.g. jobs that have been cancelled with the “scancel” command or due to exceeding the time limit) may not be accurate. It is because the total CPU time calculated for interrupted jobs may not include its descendant processes.

Resource usage of running jobs

You can check the state of a running job (including memory usage) with the sstat command:

$ sstat -j <JOB(.STEP)> -o <FIELD1,FIELD2>

where JOB(.STEP) is the unique ID of a job or jobstep (or a comma-separated list of IDs) and FIELD1,FIELD2 is a comma separated list of the required information.

To display information for all steps of a job, use the --allsteps option:

$ sstat -j <JOB> -o <FIELD1,FIELD2> –allsteps

You can list the available fields using the --helpformat option:

$ sstat --helpformat

In the following example, we list our currently running jobs (using squeue and sacct commands), and then query some statistics about one of the jobs with sstat:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           6342410       cpu     bash    alice  R       0:08      1 x1000c0s1b1n1
           6342406       cpu  testjob    alice  R       0:47      1 x1000c1s5b1n0

$ squeue --steps
         STEPID     NAME PARTITION     USER      TIME NODELIST
  6342406.batch    batch       cpu    alice      0:47 x1000c1s5b1n0
      6342410.0     bash       cpu    alice      0:08 x1000c0s1b1n1

$ sacct -o JobID%15,Partition,Account,State -j 1234567
          JobID  Partition    Account      State
--------------- ---------- ---------- ----------
        6342406        cpu   research    RUNNING
  6342406.batch              research    RUNNING

$ sstat -o JobID,MaxRSS,MaxVMSize,AveCPU,MaxDiskRead,MaxDiskWrite --allsteps -j 1234567
JobID            MaxRSS  MaxVMSize     AveCPU  MaxDiskRead MaxDiskWrite
------------ ---------- ---------- ---------- ------------ ------------
6342406.bat+   2850032K   8558440K   00:03:23   2020065913      9356795

For detailed information see the manpages for sstat (type man sstat in the terminal).

Requested vs allocated resources

In some cases, due to resource allocation constraints, Slurm will allocate more resources to your job than you requested. That will also affect the resource usage statistics.

You can query the ReqTres and AllocTres info fields of the “sacct” output to check the requested and allocated resources for a job (the %N modifier following the field name sets the width of the output):

$ sacct -j <JOBID> -o ReqTres%40,AllocTres%40

For example, the following output shows that we requested 1 cpu and 16000 MB memory but Slurm allocated 8 CPUs and 16000 MB memory. This is because the maximum amount of memory that can be requested per CPU core on the Komondor CPU partition is 2000 MB, so Slurm had to allocate 8 CPUs instead of one to provide 16000 MB of memory for the job.

$ sacct -j 1234567 -o JobID,ReqTres%45,AllocTres%45
JobID                                                        ReqTRES                                     AllocTRES
------------ ------------------------------------------------------- ---------------------------------------------
6292355                            billing=1,cpu=1,mem=16000M,node=1             billing=8,cpu=8,mem=16000M,node=1
6292355.0                                                                                  cpu=8,mem=16000M,node=1

GPU usage statistics

You can check the GPU utilization while the job is running. First, you have to interactively “connect” to the resource allocated by the running job:

$ srun --overlap --pty --jobid=<JOBID> bash

where JOBID is the unique ID of the running job.

You can now view the current GPU utilization:

$ nvidia-smi

The previous command will show all GPUs of the given node. At the bottom of the table there will be a list of processes that your job runs on any GPU. If no process is listed, your application is not using the GPU. In the table, you can view the statistics of the GPU(s) you are using.

For further information see the manpages for nvidia-smi (type man nvidia-smi in the terminal).

Job monitoring with jobstats

In addition to the built-in Slurm tools, you can monitor resource usage using jobstats, based on the Princeton University Jobstats project. We thank the original authors and contributors for the software and documentation. The Komondor integration adds Lustre I/O, NVLink throughput and short-job accounting support.

Jobstats provides both a web interface and command-line tools to inspect running and completed jobs in a more intuitive and visual way.

The Komondor cluster integrates the jobstats monitoring service. You can access the web interface at:

https://jobstats.komondor.hinfra.hu

The web interface uses Grafana to display the time series collected by Prometheus. It provides current and historical views of CPU, host memory and GPU usage per job. Use the job’s time interval to compare activity across the available panels. The time series are limited by Prometheus retention; a saved command-line summary can remain available after the underlying graph data has expired.

The monitoring data is collected automatically. Prometheus queries the compute-node CPU/GPU exporters and the admin-node Lustre exporter every 30 seconds. The admin exporter reads the Lustre storage servers’ per-job statistics; the command-line tool queries Prometheus rather than contacting those servers directly.

Example view:

• Real-time CPU usage

• Memory footprint throughout job duration

• GPU utilization if available

CLI:

$ jobstats <JOBID>

The normal report includes an online link for the job and English-language resource-efficiency notes that refer to this documentation. These notes are advice, not guarantees about queue times or future job priorities. The --simple / -s report omits general efficiency advice but retains the short-job data-source note. Use jobstats --help for the available options and Troubleshooting Resource Efficiency for the recommended troubleshooting steps.

Short jobs and missing samples

A job’s cgroup can be created and removed between two Prometheus scrapes. For completed or otherwise terminal jobs shorter than two scrape periods (currently less than 60 seconds), the text report supplements missing CPU and host-memory statistics with Slurm accounting data, using the same accounting sources as seff:

• CPU efficiency uses TotalCPU / (ElapsedRaw * AllocCPUS) * 100. A recorded zero CPU time can be a valid measurement; missing CPU time is unavailable.

• Host-memory usage uses the largest available job-step sum from TRESUsageInTot. Allocated memory comes from AllocTRES. This accounting aggregate can differ from a sampled cgroup-memory peak and from the memory request displayed in the job header.

• These are job totals, not measured per-node values. Existing usable Prometheus values are retained; only missing CPU or memory data is supplemented.

• This fallback does not estimate GPU, NVLink or Lustre values and does not change the JSON/base64 output or the stored Jobstats summary.

The report identifies the accounting source once, for example:

Short job: CPU and memory statistics are obtained from Slurm accounting.

If accounting has not yet been finalized or is unavailable, the affected values remain unavailable. A missing value must not be interpreted as zero utilization. Jobs longer than this short-job threshold can also have missing samples; the fallback is not a general replacement for missing monitoring data.

Lustre I/O statistics

On Komondor, the command-line report also includes per-job Lustre I/O statistics for the project and scratch filesystems, alongside the CPU, memory and GPU statistics. These are available for both running and completed jobs:

$ jobstats <JOBID>
$ jobstats -s <JOBID>

Both reports show a compact Lustre summary with one line per filesystem, containing the total amount of data read and written and, when available, the corresponding transfer speeds, plus the total number of exported operations. For example (illustrative values):

Lustre I/O (partial)
  project: read 245.3GB (176.2MB/s), write 253.3GB (176.2MB/s), operations 2,345
  scratch: read 76.1MB, write 251.0B, operations 37

The operations value is the sum of all exported MDT/OST operation counters for the filesystem. It includes supported read/write operation counts as well as metadata operations; the byte counters themselves are not included. It is not necessarily the number of system calls made by the application. Missing operation samples appear as operations not available; a measured zero appears as operations 0.

For a running job, the speed in parentheses is the average over the last two minutes. For a completed job, it is the average over the entire job runtime. A missing speed does not mean zero traffic: the compact report omits it when there is insufficient data to calculate the filesystem’s combined speed.

For a detailed breakdown, add --lustre-details to either report:

$ jobstats --lustre-details <JOBID>
$ jobstats -s --lustre-details <JOBID>

The detailed view separates MDT (Metadata Target) and OST (Object Storage Target) statistics. It includes read/write totals, averages over the job runtime, the last two-minute average for running jobs when available, and exported operation counts, such as MDT file opens. Data stored directly on MDTs (Data-on-MDT) also contributes to the compact I/O totals.

For machine-readable output, jobstats -j <JOBID> includes the full Lustre details in JSON format without requiring --lustre-details.

To refresh a running job’s compact report every 30 seconds, use:

$ watch -n 30 jobstats -s <JOBID>

Lustre statistics are collected automatically every 30 seconds. Even jobs shorter than 60 seconds can have I/O totals if a sample has been collected; the last two-minute transfer speed requires at least two samples.

The status in the Lustre I/O heading describes the collection coverage:

• complete: the collection coverage checks passed for the reporting period.

• partial: coverage is incomplete, for example because of collection gaps, failed sources or stale samples. Totals reflect the available samples.

• unavailable: no usable Lustre collection data is available.

Note

Lustre monitoring was introduced on Komondor on 9 October 2026. Jobs that started before monitoring began may have partial coverage, and older saved reports may contain no Lustre data. After a job finishes, allow time for its final summary to be collected and saved. Missing Lustre data does not prevent the available CPU, memory and GPU statistics from being displayed.

Note

These statistics measure traffic handled by the storage servers. Reads served from the client cache may not appear as server-side I/O, so the reported amounts can differ from the data read or written by the application.

NVLink throughput

When valid DCGM data and an unambiguous GPU-to-job mapping are available, both the normal and simple reports show average and sampled maximum NVLink throughput per GPU over the job’s reporting interval:

NVLink throughput per GPU (average / maximum)
    NODE (GPU 0): 3.96GB/s / 4.00GB/s

These values are illustrative. The source is the DCGM DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL throughput gauge. Jobstats converts its MB/s values to bytes/s and uses the CLI’s 1024-based size formatting. It averages throughput samples rather than applying rate() to this gauge. The values are neither cumulative transferred bytes nor NVLink error counts.

GPU UUIDs connect the DCGM measurements to NVML’s job attribution. The report uses the NVML GPU index, which can differ from DCGM’s device index. Attribution is checked for each sample before aggregation; throughput from a later job using the same GPU is not automatically included. Whole-device traffic is not assigned to a MIG slice.

Note

The maximum is the largest available sample, not the true peak between samples. Missing, unsupported, invalid or ambiguously attributed data is omitted rather than replaced with zero. An available zero is shown as zero. A single-GPU workload may need no NVLink traffic; a high value by itself does not establish good application performance. See Troubleshooting Resource Efficiency.

Final summaries and graphs

After a job finishes, the summarization worker waits at least 60 seconds for final collection and accounting data, then stores the available Jobstats summary in Slurm’s AdminComment field. Allow time for this process before interpreting missing fields in a newly completed job. Older stored reports may lack metrics that were added later.

The saved summary preserves aggregates, while Grafana reads time series from Prometheus. An aggregate does not recreate an expired or missing time series. For Lustre speed graphs, each point represents an average over a rolling window, not an instantaneous speed. The window must contain enough samples; a job can have a known I/O total even when a speed cannot be calculated.

Aggregated job efficiency with reportseff

For a command-line summary of job efficiency over a range of jobs, the reportseff tool is recommended.

This utility gives you an aggregated view of the efficiency metrics similar to seff, but works over many jobs and produces structured output ideal for usage analysis.

You can use it as follows:

$ reportseff <JOBID> [<JOBID2> ...]

or run it for all your jobs within a date range (eg last 4 hours):

$ reportseff --since h=4

This will generate a table or CSV-style output showing:

• CPU and memory efficiency

• GPU usage when requested and available

• Wall time and total core time

• Exit status and job state

To request the saved GPU-efficiency and GPU-memory columns explicitly:

$ reportseff --since h=4 --format=+GPUEff,GPUMem

The current reportseff does not provide Lustre or NVLink columns. Use jobstats for those aggregates and the available Grafana panels for time series.

For more information, refer to the official documentation:

• Jobstats: https://princetonuniversity.github.io/jobstats/

• reportseff: https://princetonuniversity.github.io/jobstats/tools/reportseff/

Note

reportseff queries Slurm accounting, not Prometheus. Its basic CPU, memory and time-efficiency fields can be available without a stored Jobstats summary. Where a Jobstats summary exists in AdminComment, the installed version can use it to refine CPU/memory results and obtain GPU statistics. GPU fields require usable saved GPU data; they may be missing for older jobs or while the final summary is still being prepared.