Troubleshooting Resource Efficiency

Use resource measurements to find the limiting stage of your workload and to choose an appropriate allocation. An average utilization percentage is a starting point, not a diagnosis. Different applications have different computation, I/O and synchronization patterns; there is no universal CPU or GPU utilization target.

Start with measurements

For a running or completed job:

jobstats JOBID
jobstats -s JOBID
jobstats --lustre-details JOBID

Open the online link in the normal report to inspect the available Grafana time series over the job’s interval. Compare when CPU, memory, GPU and storage activity changes rather than relying only on the whole-job average. Use seff JOBID or reportseff JOBID for completed-job Slurm accounting; for running-job accounting, see Checking Job Efficiency and sstat.

Check the allocation as well as the request:

sacct -j JOBID -o JobID,State,Elapsed,AllocCPUS,ReqTRES%50,AllocTRES%50

Memory requests can cause Slurm to allocate additional CPU cores. Compare performance against the actual allocation, not only the values in your script. For representative completed runs, record elapsed time, CPU/GPU allocation, measured memory peaks and input size. Change one resource setting at a time and compare runs with the same workload.

CPU parallelism and scaling

First establish whether the program supports parallel execution and which Programming Models it uses. Requesting more cores does not by itself make a serial program parallel. For a serial program, start with one task and one CPU core, subject to the memory allocation it needs.

For a threaded application, allocate CPUs to one task and configure the application to use them. For example, for a program built with OpenMP support:

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=1G
#SBATCH --time=00:10:00

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun ./my_openmp_program

The executable is a placeholder for your own OpenMP program. Other threading libraries may require their own settings. For MPI, --ntasks requests processes (ranks), while --cpus-per-task allocates CPUs to each process; use an MPI runtime compatible with the application’s build. For hybrid MPI/OpenMP, set both and avoid starting more threads per rank than its allocated CPUs. See OpenMP and the MPI guides under Programming Models.

Measure scaling with a small set of allocations. Compare elapsed time and total core-hours, and stop increasing cores when the extra resources no longer produce a useful improvement. A multi-node allocation requires software that distributes work across nodes; a single threaded process does not do this automatically.

If CPU usage is close to one core out of several allocated cores, check for serial execution or an incorrect thread/rank count. If CPU usage drops during parts of the run, compare storage traffic, synchronization and application logs. A low whole-job average can hide an efficient compute phase followed by a long I/O phase.

GPU activity and data supply

Verify that the software build and chosen execution mode actually support GPUs. Check the job’s application log for the selected device and backend. To inspect a running allocation interactively:

srun --overlap --pty --jobid=JOBID bash
nvidia-smi

nvidia-smi can show every GPU on a node. Identify the GPU devices and processes belonging to your allocation; an idle device that was not allocated to your job is not evidence of poor job efficiency. GPU memory allocation alone does not establish that GPU computation is occurring.

For a multi-GPU job, confirm that the application supports multiple GPUs and has been configured to use all requested devices. Frameworks may require distributed workers or an explicit parallel mode. Compare each allocated GPU’s activity and memory use rather than only the aggregate.

If GPUs wait between compute phases, investigate CPU preprocessing, data-loader workers, transfers, storage latency and synchronization. Increasing worker counts can increase host-memory usage and CPU demand; stay within the allocation and measure the effect. Batch size and model changes can affect both performance and GPU memory, so compare them against the workload’s correctness requirements.

Host memory and GPU device memory

Host RAM and GPU device memory are separate resources. Slurm’s --mem, --mem-per-cpu and --mem-per-gpu request host memory; --mem-per-gpu does not enlarge a GPU’s device memory. See Memory Allocation for the mutually exclusive request options and partition limits.

For OUT_OF_MEMORY or an application memory error, inspect the log to identify which memory resource was exhausted. Host-memory demand includes the application, containers, caches, datasets and worker processes. A GPU device-memory error may require reducing the workload per device or changing the application’s memory strategy rather than requesting more host RAM.

Use representative measured peaks with a margin for input-size changes and run-to-run variation. Sampled monitoring can miss short peaks. For multi-node jobs, the reported sum of per-node peaks need not have occurred simultaneously. Slurm accounting and cgroup measurements can also report different memory aggregates; see Checking Job Efficiency before comparing their percentages.

Lustre I/O and metadata activity

Compare project and scratch traffic against the computation phases. For a running job, the compact report’s speed is the last two-minute average; for a completed job, it is the average over the whole runtime. Neither is an instantaneous transfer speed.

A large operation count with little transferred data can warrant investigation of metadata activity, such as repeated opens, directory scans or small-file work. The compact operations total also includes supported OST read/write operations, so inspect the MDT/OST and operation breakdown with --lustre-details before drawing conclusions. Reducing repeated scans or unnecessary opens, and grouping small records where the application supports it, may help; measure a representative run after changing the I/O pattern. See Storage for storage guidance.

Lustre reports server-side activity. Client-cache reads may produce little server traffic. Incomplete collection and missing speed samples must not be interpreted as zero I/O. The complete coverage status does not remove these measurement limitations.

Runtime requests and interactive allocations

Choose --time from representative runtime measurements with a margin for variation, within the partition limit. If a job reaches TIMEOUT, inspect its progress and logs before increasing the limit. Use application checkpointing for workloads that need to survive interruption or continue across allocations.

Cancel an interactive allocation when it is no longer needed. For repeatable workflows, a batch job is often easier to size and compare. See Submitting and Managing Jobs for scancel and Interactive Use for interactive-job usage. An efficiency note does not establish a maximum permitted interactive duration, a guaranteed queue-time reduction or a future priority penalty.

Missing measurements and short jobs

Prometheus collects samples every 30 seconds. A short job or a brief utilization burst can fall between samples. Missing data is different from a valid zero; check the job state, sample coverage and data-source note before interpreting it. For terminal jobs shorter than 60 seconds, Jobstats can supplement missing CPU and host-memory values from Slurm accounting, as described in Checking Job Efficiency. This does not create per-node samples, GPU estimates or time series.

After completion, allow time for final accounting and the saved summary. Grafana can only display retained Prometheus samples; an older saved aggregate cannot reconstruct a graph. If a reproducible discrepancy remains, contact the HPC Support helpdesk with the job ID, relevant script settings, workload size and observed symptoms.