Quality of Service (QOS)

Each job submitted to Slurm will be assigned a Quality of Service (QOS). QOS determines the job’s priority (which affects how soon the job starts), specifies whether it can interrupt other jobs if there are no free resources, may set certain resource limits (e.g maximum number of CPUs, maximum running time, etc.) and can also affect billing rates (for example, twice as much or half as much CPU time is billed as used).

Important

Komondor defines the job QOSs normal, lowpri and high, plus the test-limit partition QOS for the test partition. A QOS definition existing on the cluster does not mean it is available to every account. At the check on 11 October 2026, all four had UsageFactor=1. Check your account permissions and current limits before requesting a QOS manually.

Checking Available QOSs

QOSs Defined on the Cluster

All QOS definitions that exist on the cluster can be listed with the following command:

$ sacctmgr list qos

You can customize the output by specifying a comma-separated list of the required field names as a format option. Important properties of the current QOS definitions on the Komondor are:

$ sacctmgr -n -P show qos format=Name,Priority,GraceTime,PreemptMode,UsageFactor,Preempt,MaxJobsPU
normal|1000|00:01:00|cluster|1.000000|lowpri|
lowpri|100|00:01:00|requeue|1.000000||
high|10000|00:00:00|cluster|1.000000||
test-limit|0|00:00:00|cluster|1.000000||2

This snapshot was checked on 11 October 2026. Priority affects scheduling priority; UsageFactor is the billing multiplier. Preempt lists the QOSs whose jobs this QOS can preempt, while PreemptMode describes how jobs in the QOS are preempted. A value of cluster inherits the cluster default.

The normal QOS lists lowpri in Preempt. An empty list on another QOS does not mean that its jobs are protected from preemption. MaxJobsPU=2 limits the number of simultaneously running jobs per user subject to test-limit. The Overview explains how this partition QOS applies to the test partition.

For more information on the available fields, see the “Show QOS Format Options” section in the ‘sacctmgr’ manual or see the manpages for sacctmgr (type man sacctmgr in the terminal).

Preemption on Komondor

The cluster settings checked on 11 October 2026 are PreemptType=preempt/qos, PreemptMode=REQUEUE and JobRequeue=1. Together with the QOS definitions, these allow a normal job to preempt eligible lowpri jobs when Slurm needs their resources. The scheduler decides whether and when preemption is useful; it does not imply a guaranteed start time for the pending job.

Use lowpri for workloads that can tolerate interruption. Under REQUEUE, Slurm requeues an eligible job; if it cannot be requeued, it can be cancelled. A restarted batch job runs its script again: application progress is not restored automatically. Use application checkpoint/restart support and make the script safe to repeat. Explicitly disabling requeue changes what can happen on preemption.

The GraceTime field controls the preemption grace period. Do not rely on this short interval as the only opportunity to save application state; checkpoint periodically. See the Slurm preemption documentation for signal and requeue behavior.

You can inspect the cluster-wide settings with:

scontrol show config | grep -E 'PreemptType|PreemptMode|JobRequeue'

Combine this with the QOS listing above and scontrol show partition <PARTITION> when checking a particular job’s limits and scheduling environment.

QOSs Available to Your Account

On the Komondor, currently the “lowpri” and “normal” QOSs are available to the projects by default and the default QOS (the QOS the job gets if the user did not set it explicitly) is “normal”.

You can check the QOSs that are available to your account / user and the default QOS with the following command:

$ sacctmgr list assoc account=$project format=Cluster,Account,User,QOS,DefaultQOS

   Cluster    Account       User                  QOS   Def QOS
---------- ---------- ---------- -------------------- ---------
  komondor   research                   lowpri,normal    normal
  komondor   research      alice        lowpri,normal    normal

where the value of the $project variable is the name of your project (account) and the required information is selected by adding the QOS and DefaultQOS field names to the output format specification.

Default QOS

On the Komondor, the default QOS is “normal”.

Manual QOS

You can specify a certain QOS other than the default using the --qos= option to the “sbatch”, “salloc” and “srun” commands, or as an #SBATCH directive in the sbatch script. For example, to set the QOS to “lowpri”, specify the following directive in the sbatch script:

#SBATCH --qos=lowpri

For more information on the QOS, see the “Quality of Service (QOS)” section in the Slurm documentation